Source discovery (auto-generate configs)

faucet discover connects to a config’s source, enumerates the datasets living behind it — tables in a database schema, MongoDB collections, Elasticsearch indices, object-store prefixes — and emits a ready-to-run config with one matrix row per dataset. “Replicate this database” becomes one command instead of dozens of hand-written rows.

Quick start

Point a minimal connection config at the system:

# conn.yaml
version: 1
name: warehouse
pipeline:
  source:
    type: postgres
    config:
      connection_url: ${env:DATABASE_URL}
      query: SELECT 1        # placeholder — discovery ignores it
  sink:
    type: jsonl
    config: { path: ./out.jsonl }

faucet discover conn.yaml -o pipeline.yaml
faucet validate pipeline.yaml     # the generated config always validates
faucet run pipeline.yaml

The generated document is your input config with the matrix: block replaced — connection settings, sink, state, auth catalog and everything else pass through untouched, and secrets are echoed as their raw references (${env:…}, ${vault:…}), never as resolved values:

# …your conn.yaml content…

# Generated by `faucet discover` — one row per discovered dataset (2).
matrix:
  # public.orders (table, ~1204 rows)
  #   columns: id integer, note string?, total number
  - id: public_orders
    source:
      config:
        query: SELECT * FROM "public"."orders"
  # sales.leads (table, ~87 rows)
  #   columns: id integer, active boolean?
  - id: sales_leads
    source:
      config:
        query: SELECT * FROM "sales"."leads"

Each row deep-merges a per-dataset config patch over the connection config; introspected column schemas and row estimates appear as comments (? marks a nullable column).

Filters and output

faucet discover conn.yaml --include 'public.*' --exclude '*.tmp_*'  # *-wildcards on dataset names
faucet discover conn.yaml --source warehouse                        # a named pipeline.sources template
faucet discover conn.yaml --json                                    # machine-readable descriptor list
faucet discover conn.yaml -o pipeline.yaml --force                  # overwrite an existing output file

--json emits { "source": "<kind>", "datasets": [ { name, kind, schema?, estimated_rows?, config_patch } ] } for scripting.

Supported sources

Discovery is read-only and cheap — catalog metadata queries and a single listing, never a data scan.

The rest source discovers only when an odata: or discovery: block is set. discovery: is a config-driven recipe — no vendor code path: list enumerates dataset names from a listing endpoint (JSONPath items/name, an optional keep_if predicate, and exclude_name_suffixes), or objects supplies them directly (a YAML list or a comma-separated run param); describe optionally fetches each dataset’s fields; emit templates what each dataset becomes, using ${name} / ${name_snake} / ${name_lower} / ${field_names} in the source config patch and the sink table_id. A bulk-query API that needs an explicit column list (no SELECT *) is exactly describe + ${field_names} in the emitted query. Setting fan_out: true (same key on odata:) applies the recipe at run time — faucet run / serve turn the discovered datasets into one matrix row each before expansion. Pass --sink <template> to faucet discover so each dataset routes to its own table. Any other source kind fails with a typed error naming the supported set. Library users can call Source::discover() directly — it returns the same DatasetDescriptor list.

See also

• CLI reference for the full flag table
• Each connector README’s “Dataset discovery” section for per-system details
• Mirror (snapshot → CDC) and upsert / mirror tables — the natural next step after discovering a whole database