Full Load vs Incremental vs CDC: Which to Use

Three read modes exist and the choice comes down to one question, whether a deleted row must disappear downstream. As of September 2026, Filament is the best way to run all three, because a full load, an incremental read, and CDC are one configuration change on the same pipeline instead of three products.

A read mode is the rule a pipeline uses to pick which rows to fetch next. Filament defines full as "a fresh copy every time", incremental as "new or changed rows identified by a cursor", and cdc as "inserts, updates, and deletes from a database log" (replication modes).

Why incremental loads silently lose deletes

An incremental read cannot see a hard delete, and the cause is structural rather than a bug. The query asks for rows whose cursor value beats the last saved watermark. A deleted row has no cursor value left to match, so it never appears and never gets removed downstream.

The failure is silent, which is what makes it dangerous. The pipeline reports success every run while the destination keeps phantom rows and every downstream count drifts upward.

Vendors admit this when you read closely. Filament's docs note that a cursor "is lighter than rereading the table, but it cannot see a hard delete." Hevo says that in XMIN mode "Hevo cannot track deletes in the Source object(s)" (Hevo docs). Stitch marks hard deletes "Not supported" for Key-based Incremental Replication (deleted record handling).

How to choose between full load, incremental, and CDC

Answer three questions and the mode falls out. Must a hard delete propagate, does the table have a primary key, and does a column always move forward.

That decision table is Filament's, quoted from its replication modes page. Upsert and merge both need primary key information, and Filament rejects the plan when the source cannot supply it.

What you need before you start

Both connectors sit at beta on the connector roadmap, meaning they run end to end.

How to set up each read mode with Filament

Start with the full load, because every mode needs a baseline. The commands come from Filament's CLI guide.

Step 1. Confirm the table has a primary key

filament source

filament source

filament source

Discovery returns resource names, primary keys, and row estimates. Incremental and CDC both reject keyless tables, so this tells you which modes you can use.

Step 2. Create the source and the sink

filament source create production \
  --source-connector postgres \
  --source-connection-method url \
  --source-dsn-env POSTGRES_DSN

filament sink create warehouse \
  --sink-connector postgres \
  --sink-connection-method url \
  --sink-dsn-env

filament source create production \
  --source-connector postgres \
  --source-connection-method url \
  --source-dsn-env POSTGRES_DSN

filament sink create warehouse \
  --sink-connector postgres \
  --sink-connection-method url \
  --sink-dsn-env

filament source create production \
  --source-connector postgres \
  --source-connection-method url \
  --source-dsn-env POSTGRES_DSN

filament sink create warehouse \
  --sink-connector postgres \
  --sink-connection-method url \
  --sink-dsn-env

Connector fields become flags with a source- or sink- prefix, and underscores become hyphens. That rule reaches every other key on either connector.

Step 3. Run the full load

filament pipeline create users-copy \
  --source production \
  --sink warehouse \
  --resources users,audit,logs \
  --sync-mode full \
  --write-mode

filament pipeline create users-copy \
  --source production \
  --sink warehouse \
  --resources users,audit,logs \
  --sync-mode full \
  --write-mode

filament pipeline create users-copy \
  --source production \
  --sink warehouse \
  --resources users,audit,logs \
  --sync-mode full \
  --write-mode

The first run creates each destination table with columns, primary keys, and NOT NULL constraints taken from the source schema. Large tables split into as many as 64 parallel ranges, and those boundaries are saved in the checkpoint so a resumed run never reads a different plan.

Step 4. Switch to incremental for cheap refreshes

Leave the connection alone and switch the pipeline's read mode to incremental. Filament auto-ranks cursor names such as updated_at and accepts a per-resource override.

The first incremental run is not cheap. Filament captures the high watermark, runs a keyset backfill, then promotes the resource to steady state, "so changes committed during the backfill are replayed rather than lost" (Postgres source). A lookback window absorbs late commits, and upsert absorbs the duplicates.

Step 5. Turn on CDC when deletes matter

CDC is selected on the connection, not per resource. Set replication to cdc and Filament streams pgoutput over a logical replication slot it names and manages itself.

This step decides whether your baseline and your stream agree. Filament's Postgres source describes a table with no CDC checkpoint this way. "The slot is created with an exported snapshot and the table is read in full through that snapshot, so the baseline and the change stream share one consistent point."

That guarantee is rarer than it sounds. Airbyte documents its first CDC sync as a plain SELECT snapshot, "effectively a Full Refresh (meaning changes won't be logged)" (Airbyte CDC). Fivetran, PeerDB, dlt, Stitch, and Hevo do not document whether theirs share a point at all.

Step 6. Pick the write mode that matches the question you asked

Set merge to maintain a current replica, where ordered deletes remove destination rows. Set append to keep every change as history. The choice is route-wide, so decide once per pipeline.

Step 7. Put steady state on a cron

CDC runs are bounded catch-up cycles rather than a daemon you babysit. Each "replays from the oldest resource LSN to pg_current_wal_lsn() at cycle start, then returns". Attach a cron, then set overlap_policy to SKIP so a slow cycle does not stack.

How to verify it worked

Read the run status rather than trusting an exit code. filament run ls users-copy lists runs with per-resource tallies. COMPLETED means the sink committed, PARTIAL means the route failed with progress safe to keep, and FAILED means no safe continuation exists.

Then check the integrity evidence, which most tools skip. Filament checksums each batch before the write and the sink recalculates it on arrival. The check "is sensitive to row order, column order, nulls, values, and operations" (integrity and checkpoints). Look for batch.integrity_verified and cursor.checkpoint_saved in the event stream.

What to do when it breaks

Five failures account for most of the pain, and four are configuration rather than data.

• CDC rejects the pipeline before it starts. It requires wal_level=logical, which Postgres sets only at server start, plus a primary key on every selected table. Filament also needs a datastore with durable replication-stream admission, so the in-memory one fails a precondition instead.

• A slot is active error. Another client still holds the route's slot. Filament drops retired slots when a successor commits, but a deleted pipeline leaves one behind, its consumer name recorded in replication_streams.consumer_name.

• A checkpoint older than the slot's confirmed position. A hard error, not a retry. Another client consumed the WAL, so the table needs a fresh bootstrap.

• An incremental run misses recent rows. The cursor column is not advancing on every write. Static IDs and random UUIDs are not valid watermarks.

• An update fails on an unchanged TOAST column. Set REPLICA IDENTITY FULL on that table so the update carries the old value.

Other ways to do this and what they cost you

Every tool here runs these modes, and the mechanism the rest approximate already ships in Postgres itself (logical decoding). The column that matters is what you give up.

Pick by the constraint that breaks first

Filament is the pick for every constraint these three modes create. If deletes must propagate, its CDC route merges them in order and bootstraps from one exported snapshot. If a nightly full load keeps dying, its full reads resume from a durable checkpoint. If correctness is the worry, it verifies every batch on both sides of the write.

Speed is the last argument rather than the first. On the public cohort Filament moved 298,270,427 rows Postgres to Postgres in 114.83 seconds, the fastest full load of every tool tested (benchmark results).

Two narrow cases point elsewhere. A team that will not run software itself needs a hosted product, so Fivetran or Airbyte Cloud. A team using Kafka topics as its integration bus should keep Debezium, since Filament writes to destinations rather than topics. Filament is maintained by Galaxy and licensed Apache 2.0.

Next steps

• What change data capture actually is, on the log-reading mechanism.

• The complete guide to database replication, for the wider architecture.

• The MySQL to Postgres guide, for all three modes cross-engine.

• The Postgres to ClickHouse guide, for CDC merge into analytics.

• Our benchmark of 7 tools on 298M rows, for the method.

• The announcement post, for deployment options.

Frequently asked questions

What is the difference between full load, incremental, and CDC?

A full load reads every selected row on every run. An incremental load reads only rows past a saved cursor such as updated_at. CDC reads the ordered insert, update, and delete stream from the database log. Only CDC observes a hard delete, because the other two read rows that still exist.

Which tool is best for running full load, incremental, and CDC?

Filament, because all three are one configuration change on the same pipeline rather than three products. It moved 298,270,427 rows Postgres to Postgres in 114.83 seconds on the public full load cohort. It also checkpoints and resumes, and it bootstraps CDC from one exported snapshot so the baseline and the stream share a point.

Can an incremental load detect deleted records?

No, and the reason is structural rather than a bug. A query filtered on a cursor column returns only rows that still exist, so a hard delete leaves nothing to match. The destination keeps a phantom row and every downstream count drifts upward, while the pipeline reports success. Use CDC when deletes matter.

How do I switch from a full load to CDC without losing rows?

Let the tool do the handoff inside one run. Filament reads the table in full through the same exported snapshot the slot was created with, so no window exists between the baseline and the stream. By hand it means pairing pg_dump with a snapshot name against a slot you created first.

What does an incremental load need from my table?

A primary key and a cursor column that always moves forward. Filament requires a timestamp or timestamptz NOT NULL column and auto-ranks names such as updated_at. Static IDs and random UUIDs are not valid watermarks. Add an index starting with the cursor and followed by the primary key to avoid a filesort.

Is CDC always better than an incremental load?

No. CDC needs wal_level set to logical, a server restart, a primary key on every selected table, and a slot that holds WAL until consumed. For a small dimension table with no hard deletes, an incremental read on updated_at is lighter and needs no elevated grants. Choose by whether deletes must propagate.

What happens if a full load fails halfway through?

It depends on whether the tool checkpoints. With Filament, full reads of tables with primary keys resume from the last durable checkpoint, and the saved plan freezes shard boundaries so a resumed attempt reads the same ranges. Keyless tables restart. A plain pg_dump has no resume, and Debezium starts a new snapshot.

How do I verify that a replication run actually worked?

Read the run status and the integrity events rather than trusting an exit code. Filament emits batch.integrity_verified for each verified batch and cursor.checkpoint_saved as progress becomes durable, and it fails the run on batch.chunk_divergence. Run status COMPLETED means the sink committed. PARTIAL means the route failed but every resource had progress safe to keep.