Connectors / Change data capture

Oracle CDC

source

Oracle CDC via LogMiner; committed-transaction change events, SCN bookmarks, exactly-once; needs Oracle Instant Client

Install

cargo install faucet-cli --features source-oracle-cdc
faucet init --source oracle-cdc
# Cargo.toml — use it as a library
cargo add faucet-source-oracle-cdc

Starter config

source:
  type: oracle-cdc
  config:
    # connect_string: ""    # Easy Connect (`host:1521/FREEPDB1`), TNS alias, or connect descriptor.
    # host: ""    # Database host.
    # port: 0    # Listener port.
    # service_name: ""    # Service name (e.g.
    # sid: ""    # Legacy SID.
    # username: ""    # Database user.
    # password: ""    # Password for `username`.
    # external_auth: false    # Authenticate with external credentials (OS authentication or a wallet holding the credentials) instead of `username` / `…
    # tls: { ... }    # TCPS settings applied to a `host`-form connection.
    tables: []    # REQUIRED — Tables to capture as `OWNER.TABLE`, in data-dictionary case (usually upper case).
    # start_position: { type: current }    # one of: current, earliest
      # --- Alternative variants — replace the block above with one of these ---
      # type: earliest
    # poll_interval: 1    # Seconds to wait between polls that find nothing new.
    # idle_timeout: 30    # End the fetch cycle after this long without a captured change, counted only once mining has reached the current SCN.
    # max_scn_window: 500000    # Largest SCN range mined per LogMiner session.
    # max_staged_records: 1000000    # Abort when one in-flight transaction buffers more than this many changes (bounded memory).
    # max_staged_records_total: null    # Abort when all open transactions together buffer more than this many changes, counting LOB pieces and folded updates.
    # batch_size: 1000    # `0` aggregates committed transactions into pages of up to 100,000 records, emitted at least every `idle_timeout`, each with its bookmark; otherwise each committed transaction is its own page.
    # max_connections: 2    # Maximum pooled sessions.
    # statement_timeout_secs: 600    # Per-round-trip call timeout in seconds (`0` disables).
    # flush_table: "FAUCET_LOGMNR_FLUSH"    # Table the source commits to before each mining window, forcing the log writer to flush redo up to the window's end SCN.
    # on_unsupported: null    # What to do with a change LogMiner cannot render as SQL for a tracked table (`UNSUPPORTED` rows, or a *dictionary mismatc…
    # state_key: ""    # Explicit state-store key.

Capabilities

Replay Deterministic replay
Exactly-once replay Yes
Incremental Yes
Dataset discovery No
Reports lag Yes
Compression No

Config fields

FieldTypeDefaultDescription
tables
required
array — Tables to capture as OWNER.TABLE, in data-dictionary case (usually upper case). **Required and non-empty.**
connect_string string — Easy Connect (host:1521/FREEPDB1), TNS alias, or connect descriptor. Mutually exclusive with host.
host string — Database host. Mutually exclusive with connect_string.
port integer — Listener port. Defaults to 1521 (2484 when tls.enabled).
service_name string — Service name (e.g. a PDB such as FREEPDB1). Mutually exclusive with sid.
sid string — Legacy SID. Mutually exclusive with service_name.
username string "" Database user. Required unless external_auth is set.
password string "" Password for username.
external_auth boolean false Authenticate with external credentials (OS authentication or a wallet holding the credentials) instead of username / password.
tls object enabled: falseserver_dn_match: true Transport encryption (TCPS) for the host form.
start_position object type: current Start position on a fresh run. Default current.
poll_interval integer 1 Seconds to wait between polls that find nothing new. Default 1s.
idle_timeout integer 30 End the fetch cycle after this long without a captured change, counted only once mining has reached the current SCN. Default 30s. A long-running runtime re-invokes the source to keep tailing.
max_scn_window integer 500000 Largest SCN range mined per LogMiner session. Default 500000.
max_staged_records integer 1000000 Abort when one in-flight transaction buffers more than this many changes (bounded memory). Defaults to 1,000,000; null = unbounded.
max_staged_records_total integer — Abort when all open transactions together buffer more than this many changes, counting LOB pieces and folded updates. Unset or null = unbounded.
batch_size integer 1000 0 aggregates committed transactions into pages of up to 100,000 records, emitted at least every idle_timeout, each with its bookmark; otherwise each committed transaction is its own page. Default DEFAULT_BATCH_SIZE.
max_connections integer 2 Maximum pooled sessions. Default 2.
statement_timeout_secs integer 600 Per-round-trip call timeout in seconds (0 disables). Default 600.
flush_table string FAUCET_LOGMNR_FLUSH Table the source commits to before each mining window, forcing the log writer to flush redo up to the window's end SCN. Created in the connecting user's schema when missing. Default FAUCET_LOGMNR_FLUSH.
on_unsupported object fail Handling of changes LogMiner cannot render. Default fail.
state_key string — Explicit state-store key. Derived as oracle-cdc:<database>:<table> (or a digest of several tables) when unset.
Get started

Your first pipeline runs in five minutes.

Install the CLI, scaffold a config, and move real data — nothing external to stand up.

curl -LsSf https://github.com/faucet-hq/faucet-stream/releases/latest/download/faucet-cli-installer.sh | sh
brew install faucet-hq/faucet-stream/faucet-cli