Google Cloud Storage

Install

cargo install faucet-cli --features source-gcs
faucet init --source gcs

# Cargo.toml — use it as a library
cargo add faucet-source-gcs

Starter config

source:
  type: gcs
  config:
    bucket: ""    # REQUIRED — GCS bucket name.
    # prefix: ""    # Object name prefix filter.
    # include: null    # Glob over the whole object key selecting which listed objects to read (e.g.
    # object_keys: []    # Explicit object names.
    # auth: { type: service_account_json_file }    # one of: service_account_json_file, service_account_json_inline, application_default, anonymous
      # --- Alternative variants — replace the block above with one of these ---
      # type: service_account_json_inline
      # config:
        # json: ""    # REQUIRED
      #
      # type: application_default
      #
      # type: anonymous
    # file_format: null    # Format of files stored in GCS.
    # max_objects: 0    # Hard cap on the number of objects read (after listing).
    # concurrency: 10    # Maximum concurrent object reads (default: 10).
    # batch_size: 1000    # Records per emitted `StreamPage`.
    # verify_length: true    # Verify each object's byte length against the length the store advertises, failing the read with `FaucetError::Source` on a short (truncated) or over-long transfer (#161).
    # verify_checksum: false    # Verify each object's body against the checksum the store advertises (#161).
    # max_object_bytes: 2147483648    # Largest object, once decompressed, that a format read whole (JSON array, raw text, CSV, XML, Excel, Avro, ORC, buffered Parquet, and JSON Lines under `verify_checksum`) may reach; a larger one fails with an error instead of exhausting memory.
    # max_decoded_bytes: 0    # Most bytes one object may decode to in memory where a format can expand past its input: an Avro file's records (per batch where the file streams), an Excel workbook's parts and shared strings.
    # storage_host: ""    # Optional storage-host override (e.g.
    # compression: null    # User-facing compression config.
    # csv: { ... }    # CSV dialect.
    # excel: { ... }    # Excel worksheet selection.
    # xml: { ... }    # XML record framing.
    # avro: { ... }    # Avro read/write options.
    # orc: { ... }    # ORC read options.
    # parquet: { ... }    # Parquet reader options (the `parquet:` block of an object-store source).

Capabilities

Config fields