Migrating from Meltano, Singer or Airbyte

A migration is four jobs: translate the configs, carry the bookmarks over, prove the new pipeline produces the same data, and switch without losing the way back. This page covers what is specific to arriving from a Singer-protocol setup (Meltano or plain tap | target) or from Airbyte. Writing the faucet configs themselves is the same as for any pipeline: start from faucet init --source <x> --sink <y>, read faucet schema source|sink <name>, and iterate on faucet validate.

The old pipeline stays authoritative until faucet has matched it for long enough to trust. Nothing old is deleted before that.

1. Read the old setup

Write down one line per stream: source system, stream / table, replication method and key, primary key, selected columns, mapper steps, destination table, schedule, where its bookmark lives, and what reads the table downstream. Note credentials by the name of the variable or secret that holds them, never by value.

Meltano

• select patterns are <stream>.<property> with * wildcards and a leading ! to exclude; meltano select <extractor> --list --all prints the resolved set.
• metadata per stream holds replication-method (FULL_TABLE, INCREMENTAL, LOG_BASED) and replication-key. A stream without an entry uses the tap’s default, often full table.
• Settings resolve from environment variables first, then the project .env, then the active environment, then the base plugin block. The variable for a setting is <PLUGIN_NAME>_<SETTING_NAME>, upper-cased with - turned into _ (tap-postgres / sqlalchemy_url → TAP_POSTGRES_SQLALCHEMY_URL). meltano config <plugin> list shows where each value comes from. A SQLAlchemy URL may carry a driver suffix (postgresql+psycopg2://) that a faucet connection URL does not accept, so a new variable holding a plain URL is often needed.
• Bookmarks: meltano state list names every state id; meltano state get <state_id> prints {"singer_state": {...}}. The inner singer_state object is what the tap receives as --state.

Plain Singer

The tap’s config.json holds connection settings; the catalog (catalog.json / properties.json) lists streams[] with tap_stream_id, key_properties and metadata[] (the breadcrumb: [] entry carries selected, replication-method, replication-key); state.json holds the last STATE value. Find the wrapper that runs tap | target to learn the schedule and which state file it passes.

Airbyte

Per connection, record the source and destination settings, the schedule, namespace and stream-prefix settings, and for every enabled stream its sync mode, cursor field and primary key.

Check that the destination sink lists the mode in its schema (faucet schema sink <name>); see Upsert / mirror tables.

Downstream readers

For every destination table, list what reads it and which columns it relies on, including the old tool’s metadata columns (_sdc_*, _airbyte_*) and the table naming. A change of table name, column type or metadata column breaks those readers even when the rows match.

2. Choose a replacement for each tap and target

Decide by what the tap reads, not by its name:

faucet list shows which connectors this build has; faucet list --available shows the whole registry. A target maps the same way: the native sink for the destination, or the singer sink to keep the target (vs. Meltano).

Moving from a tap to a template changes more than the runner: stream names, field shapes and per-stream sync modes are the template’s own, and credentials become template params. Compare the template’s rows (faucet hub rows) with the tap’s selected streams before committing to it.

Running a tap through the singer source

The bridge is a stepping stone. It runs the tap as a child process, so the tap and its Python runtime must be installed on every host that runs the pipeline, and throughput is the tap’s. It emits one stream per config row (records for other streams are dropped), so a multi-stream tap becomes one matrix row per stream. faucet init --source singer --discover --executable <tap> --stream <stream> runs the tap’s discovery and writes a catalog with that stream selected; without a selected stream most taps sync nothing and exit cleanly.

faucet stores the tap’s STATE value as the row’s bookmark, only after the sink confirms every record before it, and passes it back as --state. A crash replays from the last persisted STATE, and many taps re-send rows at the bookmark, so pair the bridge with a keyed sink (write_mode: upsert). See the faucet-source-singer README.

3. Translate selection and mappers

• Column selection: on a SQL source, list the columns in the query (an excluded column never leaves the database); on an API source, use the select / drop transforms.
• Stream maps become transforms: removing, renaming, casting, constant columns, fallbacks, filters, flattening and exploding all have one. Arbitrary Python expressions, date math and cross-field arithmetic do not, short of the SQL transform in builds that include it; otherwise move that logic downstream.
• Hashing: the hash transform offers SHA-256 and BLAKE3, not md5, so a hashed join key changes value. Rebuild joins on it, or exclude it from the comparison in step 6.
• PII: a mapper that existed to hide data is better expressed as a masking: block, which runs before any sink or DLQ sees a row.
• Metadata columns: Singer targets with record metadata write _sdc_extracted_at, _sdc_batched_at, _sdc_deleted_at and others. faucet’s metadata_columns: block stamps its own set under a configurable prefix (faucet schema config); a _sdc prefix keeps _sdc_extracted_at-style names, but there is no batched or deleted equivalent. Tell the owners of readers that use them.

4. Translate schedules

Meltano intervals translate to five-field cron: @hourly → 0 * * * *, @daily → 0 0 * * *, @weekly → 0 0 * * 0, @monthly → 0 0 1 * *, @yearly → 0 0 1 1 *; @once is a single faucet run. Set the schedule’s timezone explicitly to the one the old scheduler evaluated cron in. Ordered extract-loads within one job become depends_on between matrix rows.

5. Carry bookmarks over

Only for streams that will continue into tables the old pipeline already filled. A parallel run into a fresh schema usually wants a full initial load instead, so step 6 compares complete tables.

1. Stop the old schedule for that stream first; a bookmark read while it keeps running is already stale.
2. Export faucet state (faucet state export) before any change, and run each faucet state set / faucet state import with --dry-run first (Pipeline state & status).
3. Prefer a bookmark slightly behind the old one: behind re-reads a little (harmless with upsert), ahead silently skips rows.

Airbyte state blobs are not a faucet format: take the cursor value out of a stream’s state and use it as above. Never hand-edit state files or tables; faucet state set writes the versioned envelope the source expects. faucet state import --overwrite of the export from step 2 is the undo for any carry-over mistake.

6. Run side by side and compare

• Point every faucet sink at a separate schema or dataset, never at the tables the old pipeline writes, and give faucet its own state store.
• Load the full history (or backfill the same range the old tables hold), then run on schedule next to the old pipeline for several cycles, including the busiest period.
• Compare by content with faucet verify, twice: faucet against the source system (the real config), and faucet against the old pipeline’s tables with a comparison-only config whose source reads the old table and whose sink is faucet’s (cli/examples/verify_previous_pipeline_tables.yaml). Never faucet run or --repair a comparison-only config: it would copy the old rows into faucet’s table.
• Make the comparison fair: exclude both tools’ metadata columns and any column computed differently on purpose, compare during a quiet window (rows changing mid-scan show as differences), and avoid ${now.*}-windowed source queries in the comparison (everything outside the window reads as extra).

7. Cut over and keep the way back

Pick one:

• Readers move to faucet’s schema. Repoint models, views and dashboards, then stop the old schedule. Reversible by repointing back.
• faucet takes over the original tables. Stop the old schedule, carry the final bookmark over (step 5), point the sink at the original tables, validate, run once, verify, then start the schedule. Readers stay put, but the table shapes must match what they expect.

Either way, export faucet state after the first production run and watch faucet status for the first scheduled runs. Until the owner signs off, keep the old pipeline runnable (schedule disabled; project, plugins, state, connections and tables kept). An unread Postgres replication slot of an old LOG_BASED tap retains WAL on the source, so watch disk and drop it only after sign-off. A faucet run that loaded bad data into a sink with a rollback: block can be undone with faucet rollback.