Cookbook

Short recipes for one job each. Find the task, copy the config, adjust the names.

Shaping the dataTransforms, SQL, WASM and topology.4 pages Reading from sourcesPagination, auth and discovery.4 pages Moving data reliablyState, upserts, CDC, dead-letter queues, rollback.12 pages Data quality & governanceChecks, contracts, masking and policies.5 pages Config & reuseComposition, templates, secrets and tuning.7 pages Observability & lineageSLAs, profiling, lineage and costs.10 pages

Every page

Record transforms SQL transform WASM transform Topology (tee / merge / join) Pagination styles Authentication Source discovery Bearer-token REST API Incremental replication & state State & status Upsert / mirror tables Mirror: snapshot → CDC Backfill Parallel range partitioning Dead-letter queues Content verification Undoing a run Resilience File formats Compression Data-quality checks Data contracts PII detection & masking Data-flow policies Schema drift Config composition Parameters & pipeline templates Template Hub templates Secrets-manager interpolation Adaptive batching Throughput tuning Testing pipelines SLA monitoring Column profiling Notifications Lineage Dashboards & alerts Data Movement Catalog Change impact analysis Cost & usage accounting Change approvals Embedded integrations