Apache Iceberg

Install

cargo install faucet-cli --features sink-iceberg
faucet init --sink iceberg

# Cargo.toml — use it as a library
cargo add faucet-sink-iceberg

Starter config

sink:
  type: iceberg
  config:
    catalog:
      type: rest    # one of: rest, glue, sql, hms
      # uri: null    # Catalog endpoint URI.
      # warehouse: null    # Object-storage warehouse root, e.g.
      # credential: null    # REST bearer token or other catalog-specific credential.
      # properties: {}    # Arbitrary catalog properties passed through to the catalog builder and the warehouse file IO (e.g.
      # --- Alternative variants — replace the block above with one of these ---
      # type: glue
      # uri: null
      # warehouse: null
      # credential: null
      # properties: {}
      #
      # type: sql
      # uri: null
      # warehouse: null
      # credential: null
      # properties: {}
      #
      # type: hms
      # uri: null
      # warehouse: null
      # credential: null
      # properties: {}
    namespace: []    # REQUIRED — Multi-part namespace that contains the target table, e.g.
    table: ""    # REQUIRED — Name of the Iceberg table (without namespace), e.g.
    # create_table: true    # Create the table if it does not exist, inferring the schema from the first batch.
    # partition_spec: []    # Partition fields applied when creating the table.
    # write_mode: null    # Write semantics for a sink.
    # target_file_size_mb: 256    # Roll over to a new Parquet data file when the estimated file size (uncompressed Arrow bytes × 0.4) exceeds this threshol…
    # parquet: { ... }    # Parquet-level compression and encoding options.
    # snapshot_properties: {}    # Key-value pairs written into the Iceberg snapshot summary.
    # batch_size: 10000    # Maximum records buffered in memory before flushing a write to the iceberg writer pipeline.
    # cleanup_orphans_on_failure: false    # Delete the Parquet data files this flush already uploaded when the snapshot commit *definitively* fails, so they do not accumulate as orphans.

Capabilities

Config fields