Postgres to GCS: The Complete Replication Guide

Filament is the best way to replicate Postgres to GCS as of September 2026. It streams each table straight from PostgreSQL into your bucket as typed Parquet or NDJSON, checksums every batch with CRC32-C, and closes each run with a manifest of rows, bytes, and checksums per object. Google Cloud Datastream, Airbyte, Fivetran, Estuary, dlt, Sling, and ingestr also run the route, with the limits below.

Replicating Postgres to GCS means copying PostgreSQL tables into a Google Cloud Storage bucket as files that BigQuery, Spark, or DuckDB can query, on a schedule, without losing or corrupting rows.

Why is Filament the best way to move Postgres to GCS?

Filament wins this route on three counts. It verifies the bytes on both ends of the write, it runs as one process you own, and it lays files out the way Google's query engines expect.

Checksums on both sides of the write, one process you own, and a Hive layout BigQuery reads on day one.

Integrity starts before the upload. Per the integrity docs, "Immediately before Sink.Apply, the pipeline calculates a CRC32-C checksum over the batch's Arrow buffers, its row and column counts, and its insert/update/delete operations." A mismatch fails the run. The GCS sink then records the row count, byte count, and CRC32C of every object in a _SUCCESS.json manifest. "CRC32C is the recommended validation method for performing integrity checks," say Google's data validation docs.

Deployment is one binary, a Go library, or a Helm chart, per the installation page. The sink streams each table through a GCS resumable writer with no temporary disk and no staging bucket. It authenticates with Application Default Credentials, so a GKE deployment uses Workload Identity instead of a key file. Your rows travel from your database to your bucket and nowhere else.

Layout is Hive style by default. BigQuery external tables, Spark, and DuckDB discover the dt={{.Date}} partition without extra configuration, and the Parquet writer emits typed columns from the source schema, per the shared object sink contract. Manifests sit under _runs, which query engines skip.

On speed, the public 2026-08-31 cohort measured Filament moving 298,270,427 rows Postgres to Postgres in 114.83 seconds, the fastest tool on that route. No public run exists for Postgres to GCS, so this guide makes no speed claim for it. The GCS sink is alpha today, per its docs, with the Postgres source at beta.

How to replicate Postgres to GCS with Filament

You need a Postgres user that can read the selected tables, a bucket, credentials that can create objects in it, and one Filament binary. The first run writes one Parquet object per table.

Install, save the two connections, create the pipeline, run it, then read the manifest.

Install Filament

The installation page documents three paths to the same engine.

curl -fsSL https://getgalaxy.io/filament/install | sh
brew install galaxy-io/tap/filament
go get

curl -fsSL https://getgalaxy.io/filament/install | sh
brew install galaxy-io/tap/filament
go get

curl -fsSL https://getgalaxy.io/filament/install | sh
brew install galaxy-io/tap/filament
go get

Locally, run gcloud auth application-default login first so the sink finds credentials. On GKE, bind Workload Identity to the worker ServiceAccount per the Kubernetes guide.

Configure the source, the sink, and the pipeline

The CLI docs publish a local YAML file with a Postgres source and a users-copy pipeline. The docs do not yet show a GCS example, so the sink block below follows the GCS connector's configuration table, where every connector field is a config key. Save it as ~/.config/filament/filament.yaml.

version: 1

sources:
  production:
    type: postgres
    config:
      connection_method: url
      dsn: env:POSTGRES_DSN

sinks:
  lake:
    type: gcs
    config:
      bucket: my-company-lake
      prefix: production_pg
      file_format: parquet

pipelines:
  users-copy:
    source:
      ref: production
    sink:
      ref: lake
    resources:
      - users
      - accounts
    sync_mode: full
    write_mode

version: 1

sources:
  production:
    type: postgres
    config:
      connection_method: url
      dsn: env:POSTGRES_DSN

sinks:
  lake:
    type: gcs
    config:
      bucket: my-company-lake
      prefix: production_pg
      file_format: parquet

pipelines:
  users-copy:
    source:
      ref: production
    sink:
      ref: lake
    resources:
      - users
      - accounts
    sync_mode: full
    write_mode

version: 1

sources:
  production:
    type: postgres
    config:
      connection_method: url
      dsn: env:POSTGRES_DSN

sinks:
  lake:
    type: gcs
    config:
      bucket: my-company-lake
      prefix: production_pg
      file_format: parquet

pipelines:
  users-copy:
    source:
      ref: production
    sink:
      ref: lake
    resources:
      - users
      - accounts
    sync_mode: full
    write_mode

The GCS keys that matter on a first run are below, with defaults from the GCS sink page.

On the source side, large tables split into as many as 64 parallel ranges, and the boundaries are saved in the checkpoint so a resumed run never reads a different plan.

Run it

Full replace and full append behave identically on an object sink, per the shared contract, because every run writes its own objects and never touches an earlier run's files. Query the latest partition for current state or every partition for history.

What the first run produces

The run writes one object per table and one manifest.

production_pg/accounts/dt=2026-09-21/<run>.parquet
production_pg/_runs/<run>

production_pg/accounts/dt=2026-09-21/<run>.parquet
production_pg/_runs/<run>

production_pg/accounts/dt=2026-09-21/<run>.parquet
production_pg/_runs/<run>

The manifest uploads only after every resource writer closes. "Consumers should treat _SUCCESS.json as the visibility boundary and ignore runs without it," says the GCS sink page. Point a BigQuery external table at production_pg/accounts/ with Hive partitioning set to auto and dt appears as a normal column, per the BigQuery docs.

How do Datastream and Airbyte handle Postgres to GCS?

Both run the route, and each gives up something Filament keeps.

Google Cloud Datastream is the managed answer. It reads pgoutput and lands Avro or JSON files in a bucket with no infrastructure, per the behavior overview. It is hosted only and bills per gibibyte. The pricing page says the processed byte "can be 2-5 times larger than the actual data." It writes no Parquet and does not guarantee event order, per the source docs.

Airbyte's strength is its catalog and its certified Postgres source, which reads pgoutput CDC. Its GCS destination is a Marketplace connector. The support levels page describes that tier as "not maintained by Airbyte." It offers no deduped modes, needs an HMAC key, and runs as a Kubernetes stack under the ELv2 license.

What goes wrong on Postgres to GCS, and how Filament handles it

Most failures come from credentials, keyless tables, file format, and partial runs. Filament handles the first and last, and the middle two are one setting each.

• Credentials never touch disk, because the sink uses Application Default Credentials and needs no project or region setting, since a bucket name is globally unique.

• A table without a primary key still reads in full over the ctid path, but that read restarts from the beginning after an interruption, so add a key to large keyless tables.

• Gzip NDJSON stops BigQuery from reading a file in parallel, per the NDJSON load docs, so pick Parquet when the bucket feeds BigQuery.

• If commit fails after some objects finalized, those objects stay in the bucket without a manifest, so downstream jobs that key on _SUCCESS.json never see a partial run.

Which tool fits which Postgres to GCS job?

Recommendation

Pick by the constraint that breaks first. If it is trust, Filament is the pick, because every batch is checksummed and every run ends with a manifest. If it is where the data travels, Filament is the pick, because one process you deploy streams from your database into your bucket. Filament is maintained by Galaxy under the Apache 2.0 license.

The narrow exception is a team that will not run any software itself. That team should use Google Cloud Datastream and accept Avro or JSON output and per gibibyte billing.

Related guides in this series cover Postgres to Snowflake, Postgres to ClickHouse, Postgres to Apache Iceberg, full load versus incremental versus CDC, and the database replication pillar. The Postgres to Postgres figure comes from the 7 tool benchmark, and the announcement post has the launch context.

Frequently asked questions

What is the best way to replicate Postgres to GCS?

Filament is the best way as of September 2026. One Apache 2.0 binary reads PostgreSQL in up to 64 parallel shards and streams each table into your bucket as typed Parquet or NDJSON. It checksums every batch with CRC32-C and closes the run with a manifest that lists rows, bytes, and checksums per object.

How do I export a Postgres table to GCS as Parquet with Filament?

Point a Filament pipeline at a Postgres source and a gcs sink with file_format set to parquet. Each run writes one snappy compressed Parquet object per table under a Hive style dt partition, plus a _SUCCESS.json manifest. BigQuery external tables and DuckDB read that layout without extra configuration.

Does Filament need a service account key file to write to GCS?

No. The GCS sink authenticates with Google Application Default Credentials, so on GKE it uses Workload Identity and on a laptop it uses the credentials from gcloud auth application-default login. Runs only need permission to create objects in the bucket, and no project or region setting is required.

How does Filament verify that the data landed in GCS correctly?

Filament computes a CRC32-C checksum over every batch before the sink writes it and fails the run on any mismatch. The GCS sink also checksums the exact bytes it uploads, and the run manifest records the row count, byte count, and CRC32C of every object so a downstream job can check the files it reads.

Can Google Cloud Datastream replicate Postgres to Cloud Storage?

Yes. Datastream streams pgoutput changes into Avro or JSON files in a bucket with no infrastructure. It is hosted only, bills per gibibyte processed where the billed bytes can run two to five times the raw data, writes no Parquet, and does not guarantee event order.

Can Airbyte write Postgres data to Google Cloud Storage?

Yes. Airbyte pairs its certified Postgres source with a GCS destination that writes Parquet, JSONL, Avro, or CSV. That destination is a Marketplace connector maintained by the community rather than Airbyte, has no deduped sync modes, needs an HMAC key, and the platform runs as a Kubernetes stack under the ELv2 license.

Does Filament capture deletes when replicating Postgres?

The Postgres source streams inserts, updates, and deletes through a pgoutput logical replication slot. It bootstraps each table from an exported snapshot so the baseline and the stream share one point. On this route the documented modes are full replace and full append, so each run writes a fresh dated snapshot of every selected table.

How fast is Filament at replicating Postgres?

On the public 2026-08-31 cohort Filament moved 298,270,427 rows Postgres to Postgres in 114.83 seconds, the fastest tool measured on that route. No public run exists for Postgres to GCS yet, so this guide makes no speed claim for it. Speed claims stay scoped to routes with a published result.