Postgres to Apache Iceberg: The Complete Guide

Postgres data reaches Apache Iceberg four ways, a scripted batch export, a replication tool that writes Iceberg directly, a Debezium or Flink CDC pipeline, or a managed service. What decides between them is whether deletes must propagate, how fresh the lake must be, which catalog your engines trust, and who pays for the compute.

Apache Iceberg is an open table format that tracks immutable Parquet files through snapshots and a catalog, so any engine can read the same table. As of October 2026 only two tools have a public measured result on this route at scale, and the gap between them is about 13 percent on wall time. The criteria below matter more than raw speed.

What decides how to sync Postgres to Iceberg?

Five criteria separate the approaches, and the first two eliminate most tools before you look at speed.

• Change capture. If deleted rows must disappear from the lake, you need logical replication, because an updated_at watermark cannot see a hard delete.

• Delete handling in Iceberg. Data files are immutable, so the Iceberg spec "adds delete files to encode rows that are deleted in existing data files" in format version 2 and deletion vectors in version 3. The tool must write one of those or rewrite the table.

• Catalog support. The tool must commit through the catalog your query engines use, whether that is Glue, a REST catalog such as Polaris, Lakekeeper, or Nessie, Unity Catalog, or Snowflake Open Catalog.

• Resume and integrity. A multi-hour load across hundreds of millions of rows will fail at least once, so checkpoints and a verifiable write matter more than peak throughput.

• Cost and footprint. The bill is either compute you already run or a meter on rows or gigabytes, and the footprint is one process or a Kafka cluster plus a Connect cluster plus a JVM.

What are the ways to replicate Postgres to Iceberg?

Every tool below can write Iceberg tables from Postgres. Two popular Postgres replication tools cannot. PeerDB deprecated every destination except ClickHouse and Postgres in July 2026 and never had an Iceberg sink, and ClickPipes targets ClickHouse Cloud only.

OLake is the Iceberg specialist. Its README lists Glue, Hive, JDBC, and REST catalogs including Nessie, Polaris, Unity, and Lakekeeper, the broadest list here, and it posted the shortest wall time on the public 298 million row run. Its writes go through a Java sidecar over gRPC, which is where its CPU bill comes from. Its licence file is Apache 2.0.

Debezium Server with the Iceberg sink is the mature streaming path. The Debezium docs describe a sink that "consumes database change events from a source Debezium connector and writes them to Apache Iceberg tables," in upsert or append mode. The community memiiso project shipped a release on October 4, 2026 with deletion vectors as the default. You run a JDK process and own its tuning.

Flink CDC is the same idea on a Flink cluster. The Iceberg pipeline connector creates tables and supports primary-key upsert, but the Postgres source cannot synchronize table structure changes.

Airbyte's S3 Data Lake destination supports REST, Glue, Nessie, and Polaris and deduplicates with an equality delete followed by an insert. It is licensed under ELv2, not an open source license, and it stores numeric primary keys as strings.

Estuary Flow runs real CDC into Iceberg, but its Iceberg materialization "orchestrates Spark jobs running on your configured compute system," meaning your own EMR Serverless application. Pricing is $0.50 per GB plus $100 per connector, and the code is BSL.

Fivetran's Managed Data Lake Service writes Iceberg to S3, ADLS, or GCS behind a Polaris-based catalog any REST client can read. Usage is metered in monthly active rows, and Fivetran warns you not to modify its tables through any external catalog.

Sling, dlt, and Snowflake cover narrower cases. Sling loads Postgres to Iceberg in one CLI line but gates parallel streams and CDC behind paid tiers under GPLv3. dlt writes Iceberg through PyIceberg with no Postgres CDC source. Snowflake's Openflow connector writes Snowflake-managed Iceberg tables in preview, which fits when Snowflake is already the lake.

Where does Filament fit on Postgres to Iceberg?

Filament by Galaxy fits when you want one open source process you run yourself, with checkpoints and a verified write, on a cron-driven CDC cadence. Galaxy builds Filament, so weigh this section against the criteria above.

On change capture, the Postgres source streams pgoutput over a logical replication slot. It bootstraps each table from an exported snapshot, "so the baseline and the change stream share one consistent point." On delete handling, the Iceberg sink accepts replace, append, upsert, incremental delete, and CDC merge. Primary keys become Iceberg identifier fields. The docs do not say which delete encoding it writes, so plan compaction as for any writer.

On catalogs, the sink ships REST, Polaris, and Lakekeeper providers plus a generic provider for any iceberg-go catalog type, and it writes data files to s3, gs, or azure storage. The sink sits on the roadmap as an alpha connector, the earliest maturity label, although it ran the public benchmark end to end. The Postgres source is beta.

On resume and integrity, the integrity docs state that "the pipeline calculates a CRC32-C checksum over the batch's Arrow buffers" and the sink recalculates it before commit. Large tables split into as many as 64 parallel ranges with boundaries frozen in the checkpoint. One limit applies. Iceberg is a commit-durable sink, so a failed full load restarts rather than resuming.

On cost and footprint, there is no per-row meter and no Java sidecar. The numbers below come from the 2026-08-31 cohort, the only public Postgres to Iceberg run at this scale, with memory and CPU from the median repetition. OLake won on wall time and Filament won on resources, and full loads say nothing about CDC throughput.

The ordering flips on the smaller TPC-H load, so table size decides who is quicker. The result files in the benchmarks repository win over any summary, including this one.

How to replicate Postgres to Iceberg with Filament

Install the binary, declare a Postgres source and an Iceberg sink in YAML, validate, then run a full load before you turn on CDC. The CLI docs describe a local context that "runs Filament in process with file-backed SQLite," which is enough for full and incremental syncs. Install with the script from the installation page, or with Homebrew.

curl -fsSL https://getgalaxy.io/filament/install | sh

curl -fsSL https://getgalaxy.io/filament/install | sh

curl -fsSL https://getgalaxy.io/filament/install | sh

Write the connections and the pipeline to ~/.config/filament/filament.yaml. The docs contain no complete Postgres to Iceberg sample, so this file is assembled from the documented keys on the source and sink pages and the REST catalog shape the public benchmark used. Validate it before trusting it.

version: 1

sources:
  production:
    type: postgres
    config:
      connection_method: url
      dsn: env:POSTGRES_DSN

sinks:
  lake:
    type: iceberg
    config:
      catalog:
        provider: rest
        uri: https://catalog.example.com/api/catalog
        warehouse: s3://my-bucket/warehouse
        auth:
          type: client_credentials
          client_id: env:ICEBERG_CLIENT_ID
          client_secret: env:ICEBERG_CLIENT_SECRET
        properties:
          s3.region: us-east-2
      namespace: analytics

pipelines:
  orders-to-lake:
    source:
      ref: production
    sink:
      ref: lake
    resources:
      - orders
      - customers
    sync_mode: full
    write_mode

version: 1

sources:
  production:
    type: postgres
    config:
      connection_method: url
      dsn: env:POSTGRES_DSN

sinks:
  lake:
    type: iceberg
    config:
      catalog:
        provider: rest
        uri: https://catalog.example.com/api/catalog
        warehouse: s3://my-bucket/warehouse
        auth:
          type: client_credentials
          client_id: env:ICEBERG_CLIENT_ID
          client_secret: env:ICEBERG_CLIENT_SECRET
        properties:
          s3.region: us-east-2
      namespace: analytics

pipelines:
  orders-to-lake:
    source:
      ref: production
    sink:
      ref: lake
    resources:
      - orders
      - customers
    sync_mode: full
    write_mode

version: 1

sources:
  production:
    type: postgres
    config:
      connection_method: url
      dsn: env:POSTGRES_DSN

sinks:
  lake:
    type: iceberg
    config:
      catalog:
        provider: rest
        uri: https://catalog.example.com/api/catalog
        warehouse: s3://my-bucket/warehouse
        auth:
          type: client_credentials
          client_id: env:ICEBERG_CLIENT_ID
          client_secret: env:ICEBERG_CLIENT_SECRET
        properties:
          s3.region: us-east-2
      namespace: analytics

pipelines:
  orders-to-lake:
    source:
      ref: production
    sink:
      ref: lake
    resources:
      - orders
      - customers
    sync_mode: full
    write_mode

Secrets stay out of the file through the env: prefix, and the sink's TestConnection loads the catalog and lists namespaces, so a bad token fails fast. Then validate, discover, and run.

filament config validate
filament source discover production
filament run orders-to-lake
filament run ls

filament config validate
filament source discover production
filament run orders-to-lake
filament run ls

filament config validate
filament source discover production
filament run orders-to-lake
filament run ls

The first run creates the namespace and tables as Parquet and maps every Postgres type to an Iceberg primitive, with arrays landing as text. New source columns are appended on later runs, because schema evolution is add-only and type changes or drops are never applied.

To move to CDC, set replication: cdc on the source and give the pipeline sync_mode: cdc with write_mode: merge. The database needs wal_level=logical and a primary key on every selected table, and the source maintains a publication named filament by default. CDC runs need the PostgreSQL datastore rather than local SQLite, so this is where you move to the Helm deployment or a hosted standalone image.

Each CDC run is a bounded catch-up cycle to the current WAL position on a cron, so freshness equals your cron interval. Verify a load by running SELECT count(*) FROM orders in Postgres and the same count against analytics.orders in your query engine.

What goes wrong when replicating Postgres to Iceberg?

Most failures on this route are replication slot hygiene and Iceberg file hygiene, not tool bugs.

• Replication slots hold WAL forever by default. The Postgres docs note that with the default max_slot_wal_keep_size slots "may retain an unlimited amount of WAL files," so a stalled consumer fills the disk. Set a cap and drop slots you stop using.

• Updates and deletes need a replica identity. A published table must have a replica identity to replicate them, and tables with TOAST columns often need REPLICA IDENTITY FULL so old values travel with the change.

• DDL does not replicate. Logical replication does not carry schema changes, so a dropped or retyped column has to be handled in the lake by hand, and a TRUNCATE on a tracked table fails a Filament CDC run.

• Frequent commits mean frequent compaction. Every cycle adds a snapshot and files, so schedule rewriteDataFiles and snapshot expiry from day one, as the Iceberg maintenance docs advise.

Which option fits which job?

Pick by the constraint that breaks first. For a team that wants an open source engine inside its own network, with CRC-verified writes and CDC merge into Iceberg on a cron, Filament fits. The sink's early maturity label and the cron-bound cadence are the two facts to weigh against it.

Related guides

See the sibling routes Postgres to Snowflake, Postgres to ClickHouse, and Postgres to GCS. For the change capture side, read Postgres CDC methods compared and the Postgres logical replication guide. The full cohort behind the numbers above is in Postgres replication benchmark, 7 tools tested on 298M rows, and the engine was introduced in Introducing Filament.

Frequently asked questions

Can you replicate Postgres to Iceberg without Kafka?

Yes. Debezium Server writes change events straight to Iceberg tables with no Kafka in between, and OLake, Filament, Estuary, Sling, dlt, and the Snowflake Openflow connector all read Postgres and write Iceberg as one job. Kafka plus Kafka Connect is only required when you want a shared event bus that other consumers also read.

How do Postgres updates and deletes land in Iceberg?

Iceberg data files are immutable, so an update or delete is recorded in a separate delete structure rather than rewriting the file. Format version 2 uses position or equality delete files, and version 3 uses binary deletion vectors. The replication tool chooses which it writes, and the query engine applies them at read time until compaction rewrites the files.

Which Iceberg catalog should I use for Postgres replication?

Pick the catalog your query engines already trust. AWS Glue fits an AWS-only lake, Apache Polaris, Lakekeeper, and Nessie are open REST catalogs you can self-host, and Unity Catalog and Snowflake Open Catalog expose the Iceberg REST API for their platforms. Most replication tools speak REST, so a REST-capable catalog keeps the most options open.

Do I need wal_level set to logical to sync Postgres to Iceberg?

Only for change data capture. A full or incremental copy works with the default replica setting because it reads tables with ordinary queries. CDC decodes the write-ahead log through a replication slot, which requires the logical level, a server restart to change it, and a primary key or replica identity on every published table.

What is the small files problem when replicating into Iceberg?

Every commit to an Iceberg table creates a new snapshot and usually new data files. A CDC pipeline that commits every few minutes produces thousands of small Parquet files and a growing metadata tree, which slows queries and bloats manifests. The fix is scheduled compaction with rewriteDataFiles and regular snapshot expiry, which the Iceberg maintenance docs recommend for any frequently written table.

What is the fastest way to move Postgres to Iceberg?

On the only public run that exists, the 2026-08-31 cohort, OLake finished the 298 million row load about 13 percent sooner than Filament on wall time. It spent more than an order of magnitude more CPU seconds to do so. No other tool has published a Postgres to Iceberg result at that scale, so any speed claim beyond those two is untested.

Does Filament support CDC from Postgres into Iceberg?

Yes. The Postgres source streams pgoutput over a logical replication slot and the Iceberg sink accepts CDC append and merge, so inserts, updates, and deletes can be applied by primary key. CDC runs are bounded catch-up cycles on a cron rather than an always-on stream, and they need the PostgreSQL datastore rather than the local SQLite context.

Which Iceberg catalogs does Filament support?

The Iceberg sink has first-class providers for REST, Apache Polaris, and Lakekeeper, plus a generic provider that passes any iceberg-go catalog type such as sql, hive, or glue straight through. Authentication can be none, a bearer token, or OAuth client credentials. Data files go to S3, GCS, or Azure storage through the catalog's FileIO.