Google Cloud Storage

Install

cargo install faucet-cli --features sink-gcs
faucet init --sink gcs

# Cargo.toml — use it as a library
cargo add faucet-sink-gcs

Starter config

sink:
  type: gcs
  config:
    bucket: ""    # REQUIRED — GCS bucket name.
    # prefix: ""    # Object-name prefix for written files.
    # path: ""    # Object key template inside the bucket, after `prefix` — the file sink's `path` (#777).
    # if_exists: null    # What happens when a file this run writes already exists (`if_exists`).
    # write_mode: null    # Pipeline-level write mode (`write_mode`).
    # format: null    # On-the-wire format of objects written by the GCS sink.
    # auth: { type: service_account_json_file }    # one of: service_account_json_file, service_account_json_inline, application_default, anonymous
      # --- Alternative variants — replace the block above with one of these ---
      # type: service_account_json_inline
      # config:
        # json: ""    # REQUIRED
      #
      # type: application_default
      #
      # type: anonymous
    # file_extension: ".jsonl"    # File extension for written objects (default `.jsonl`).
    # max_records_per_file: 0    # Hard cap on records per uploaded object.
    # max_bytes_per_file: 0    # Maximum **bytes** per object before rolling to a new one (#618).
    # concurrency: 10    # Maximum number of uploads in flight (default 10): a page that rolls into several objects uploads them concurrently while the next object is encoded.
    # batch_size: 1000    # Records per object when `max_records_per_file` is not set (without `path`).
    # storage_host: ""    # Optional storage-host override (integration-test escape hatch).
    # compression: null    # User-facing compression config.
    # csv: { ... }    # CSV dialect.
    # excel: { ... }    # Excel worksheet selection.
    # xml: { ... }    # XML record framing.
    # avro: { ... }    # Avro read/write options.
    # parquet: { ... }    # Parquet write options.
    # json_lines: { ... }    # JSON Lines write options.
    # encryption: null    # Encrypt objects at rest (#777; the `encryption` feature).
    # scratch_dir: ""    # Directory the sink builds each object in before uploading it (default: the system temporary directory).

Capabilities

Config fields