Apache Iceberg

Install

cargo install faucet-cli --features source-iceberg
faucet init --source iceberg

# Cargo.toml — use it as a library
cargo add faucet-source-iceberg

Starter config

source:
  type: iceberg
  config:
    catalog:
      type: rest    # one of: rest, glue, sql, hms
      # uri: null    # Catalog endpoint URI.
      # warehouse: null    # Object-storage warehouse root, e.g.
      # credential: null    # REST bearer token or other catalog-specific credential.
      # properties: {}    # Arbitrary catalog properties passed through to the catalog builder and the warehouse file IO (e.g.
      # --- Alternative variants — replace the block above with one of these ---
      # type: glue
      # uri: null
      # warehouse: null
      # credential: null
      # properties: {}
      #
      # type: sql
      # uri: null
      # warehouse: null
      # credential: null
      # properties: {}
      #
      # type: hms
      # uri: null
      # warehouse: null
      # credential: null
      # properties: {}
    table: ""    # REQUIRED — Table to read, as `namespace.table` (multi-level namespaces are dot-separated: `lake.analytics.events`).
    # columns: []    # Columns to read (projection).
    # filter: null    # Row filter pushed down to the scan, e.g.
    # snapshot_id: null    # Time travel: read this snapshot id instead of the current one.
    # as_of_timestamp: null    # Time travel: read the snapshot that was current at this RFC 3339 instant (e.g.
    # batch_size: 1000    # Arrow batch size (rows per page).
    # concurrency: 4    # Data files read concurrently.
    # mode: null    # How much of the table each run reads.
    # on_rewrite: null    # What an incremental run does when a snapshot since the bookmark rewrote existing data (`overwrite` / `delete`), or the b…
    # on_expired: null    # What an incremental run does when the bookmarked snapshot (or one between it and the current snapshot) has been expired …

Capabilities

Config fields