Skip to content
faucetstream

Source discovery (auto-generate configs)

faucet discover connects to a config’s source, enumerates the datasets living behind it — tables in a database schema, MongoDB collections, Elasticsearch indices, object-store prefixes — and emits a ready-to-run config with one matrix row per dataset. “Replicate this database” becomes one command instead of dozens of hand-written rows.

Quick start

Section titled “Quick start”

Point a minimal connection config at the system:

conn.yaml
version: 1
name: warehouse
pipeline:
  source:
    type: postgres
    config:
      connection_url: ${env:DATABASE_URL}
      query: SELECT 1        # placeholder — discovery ignores it
  sink:
    type: file
    config: { path: ./out.jsonl }
Terminal window
faucet discover conn.yaml -o pipeline.yaml
faucet validate pipeline.yaml     # the generated config always validates
faucet run pipeline.yaml

The generated document is your input config with the matrix: block replaced — connection settings, sink, state, auth catalog and everything else pass through untouched, and secrets are echoed as their raw references (${env:…}, ${vault:…}), never as resolved values:

# …your conn.yaml content…


# Generated by `faucet discover` — one row per discovered dataset (2).
matrix:
  # public.orders (table, ~1204 rows)
  #   columns: id integer, note string?, total number
  - id: public_orders-1f58f903
    legacy_state_ids: [public_orders]
    source:
      config:
        query: SELECT * FROM "public"."orders"
    sink:
      config:
        path: ./out.public_orders-1f58f903.jsonl
  # sales.leads (table, ~87 rows)
  #   columns: id integer, active boolean?
  - id: sales_leads-befd6242
    legacy_state_ids: [sales_leads]
    source:
      config:
        query: SELECT * FROM "sales"."leads"
    sink:
      config:
        path: ./out.sales_leads-befd6242.jsonl

Each row deep-merges a per-dataset config patch over the connection config; introspected column schemas and row estimates appear as comments (? marks a nullable column).

The row id is also the row’s state key, so it depends only on the dataset itself — never on which other datasets a listing returns. A name that is already a valid row id (letters, digits, _, -) and not reserved is used as is (orders → orders); any other name is collapsed to _ runs and gets a stable -<hash> of the full name (public.orders → public_orders-1f58f903), so a later dataset that sanitizes alike (public/orders) cannot rename it.

Releases before this rule kept a sanitized name plain while no other dataset shared it (public.orders → public_orders). Where that old id differs, the row lists it under legacy_state_ids: while the row has no state of its own, its first run moves the bookmark stored under the old id to the new one. The same happens for rows that runtime fan-out (fan_out: true) generates.

When the sink writes one fixed file — a file / jsonl / csv / single-file parquet path, or an s3 / gcs / azure-blob sink with a fixed path (sftp: file_name) — every row would overwrite the others, and faucet run refuses rows that share a destination. So each row gets its own: the row id goes into the file name before its first extension (./out.jsonl → ./out.public_orders-1f58f903.jsonl, part-{part}.parquet → part-{part}.orders.parquet), or becomes a subdirectory of a directory path (exports/ → exports/orders/). A sink that already gives every row its own name (an appending jsonl / csv, a remote sink without path, a table-per-dataset sink_patch) is left as is.

Filters and output

Section titled “Filters and output”
Terminal window
faucet discover conn.yaml --include 'public.*' --exclude '*.tmp_*'  # *-wildcards on dataset names
faucet discover conn.yaml --source warehouse                        # a named pipeline.sources template
faucet discover conn.yaml --json                                    # machine-readable descriptor list
faucet discover conn.yaml -o pipeline.yaml --force                  # overwrite an existing output file

--json emits { "source": "<kind>", "datasets": [ { name, kind, schema?, estimated_rows?, config_patch } ] } for scripting.

Supported sources

Section titled “Supported sources”

Discovery is read-only and cheap — catalog metadata queries and a single listing, never a data scan.

Source Lists Schema Row estimate Row patch
postgres base tables (all non-system schemas) information_schema.columns pg_class.reltuples query
mysql base tables (current database) information_schema.columns table_rows query
mssql base tables INFORMATION_SCHEMA.COLUMNS sys.partitions query
sqlite tables (sqlite_master) pragma_table_info — query
mongodb collections (non-system.*) inferred from a 10-doc sample estimated_document_count collection
elasticsearch indices (non-.-system) and data streams _mapping field types _cat/indices docs.count (indices only) index
bigquery dataset.table (physical tables; capped at 500 enumerated / 100 schema fetches, warned) tables.get numRows query
snowflake schema.table (base tables) information_schema.columns row_count query
spanner base tables (default schema) INFORMATION_SCHEMA.COLUMNS — query
s3 common prefixes under the configured prefix (one delimiter listing; falls back to per-object entries) — — prefix
gcs same as s3 — — prefix (objects: object_keys)
file every readable file under path (the listing the run would read; no file opened) — — path + the resolved format
oracle tables (ALL_TABLES) ALL_TAB_COLUMNS — query (native JSON columns wrapped in JSON_SERIALIZE)
iceberg tables in every catalog namespace (nested namespaces walked) the table’s current Iceberg schema snapshot summary total-records table
dynamodb tables (ListTables) key attributes only (DescribeTable key schema — DynamoDB is schemaless beyond the key) ItemCount table_name
rest + odata: OData entity sets (from $metadata EDMX) EDM property types — odata.entity
rest + discovery: whatever the recipe’s listing endpoint returns per-dataset describe request (optional) — emit.config (templated; + optional sink table_id / sink_ref)

The rest source discovers only when an odata: or discovery: block is set. discovery: is a config-driven recipe — no vendor code path: list enumerates dataset names from a listing endpoint (JSONPath items/name, an optional keep_if predicate, exclude_name_suffixes, and next — a JSONPath to the next listing page’s URL for a paginated listing), or objects supplies them directly (a YAML list or a comma-separated run param); describe optionally fetches each dataset’s fields; emit templates what each dataset becomes, using ${name} / ${name_snake} / ${name_lower} / ${field_names} in the source config patch and the sink table_id. A bulk-query API that needs an explicit column list (no SELECT *) is exactly describe + ${field_names} in the emitted query. Setting fan_out: true (same key on odata:) applies the recipe at run time — faucet run / serve turn the discovered datasets into one matrix row each before expansion, and faucet schedule does so on every tick. Such a config may not also carry a matrix:, and only one source may set fan_out: true. Pass --sink <template> to faucet discover so each dataset routes to its own table. Any other source kind fails with a typed error naming the supported set. Library users can call Source::discover() directly — it returns the same DatasetDescriptor list.

See also

Section titled “See also”