Using Testcontainers at Galaxy: Beyond mocking

When I first started building software I was obsessed with unit tests. I wouldn’t ship code without at least 80% unit test coverage. I was meticulous with mocking, ensuring I could test every piece of logic even the ones with external dependencies. As an EM I would enforce this in CI checks and leave comments to add some tests.

Then Mitch would just send this meme…

Theres a lot of truth to this meme, but that doesn’t mean all tests are useless. Of course, unit tests alone won’t prevent outages, bugs, or disasters like the Titanic, but having comprehensive test suites is crucial especially for open source software.

The problem is that not everything worth testing can be mocked.

We ran into this problem while building filament, our data movement engine. There are plenty of unit-testable components in filament, but the real meat and potatoes of data movement live in remote systems. You can't build a fixture sophisticated enough to truly behave like a PostgreSQL database or a ClickHouse instance.

We could manage these dependencies with Docker Compose, but Testcontainers lets us define the complete environment, its version, configuration, readiness conditions, and lifecycle in the test itself.

Our founding engineer Atterpac introduced Testcontainers while building the ingestion engine and had already written a few integration and end-to-end tests with it. I was immediately obsessed with the ergonomics of the project.

Here is the basic idea, using PostgreSQL:

ctx := t.Context()

pgContainer, err := postgres.Run(
	ctx,
	"postgres:16-alpine",
	postgres.WithInitScripts(filepath.Join("testdata", "init-db.sql")),
	postgres.WithDatabase("test-db"),
	postgres.WithUsername("postgres"),
	postgres.WithPassword("postgres"),
	postgres.BasicWaitStrategies(),
)
if err != nil {
	t.Fatal(err)
}
testcontainers.CleanupContainer(t, pgContainer

ctx := t.Context()

pgContainer, err := postgres.Run(
	ctx,
	"postgres:16-alpine",
	postgres.WithInitScripts(filepath.Join("testdata", "init-db.sql")),
	postgres.WithDatabase("test-db"),
	postgres.WithUsername("postgres"),
	postgres.WithPassword("postgres"),
	postgres.BasicWaitStrategies(),
)
if err != nil {
	t.Fatal(err)
}
testcontainers.CleanupContainer(t, pgContainer

ctx := t.Context()

pgContainer, err := postgres.Run(
	ctx,
	"postgres:16-alpine",
	postgres.WithInitScripts(filepath.Join("testdata", "init-db.sql")),
	postgres.WithDatabase("test-db"),
	postgres.WithUsername("postgres"),
	postgres.WithPassword("postgres"),
	postgres.BasicWaitStrategies(),
)
if err != nil {
	t.Fatal(err)
}
testcontainers.CleanupContainer(t, pgContainer

Dead simple, but extremely valuable.

How we use Testcontainers in Filament

Starting a container is the easy part. The useful work is building a test harness that makes
real infrastructure repeatable, isolated, and fast enough to run in CI.

Filament has three test layers:

1. Unit tests run without Docker

2. Integration tests exercise a component against a real dependency

3. End-to-end tests cross a production boundary

   1. Compiled process, HTTP, NATS, Kubernetes, etc.

Testcontainers powers the latter two test layers, but just using a container does not automatically make a test truly end-to-end. The boundary under test is what matters.

Reusable infrastructure helpers

We wrap each dependency in a helper rather than configuring containers in every test:

pg := testcontainers.Postgres(t)
pool := pg.Pool()
dsn := pg.DSN

pg := testcontainers.Postgres(t)
pool := pg.Pool()
dsn := pg.DSN

pg := testcontainers.Postgres(t)
pool := pg.Pool()
dsn := pg.DSN

The helper selects the image, waits for readiness, creates a connection pool, and registers cleanup with t.Cleanup .

We have similar helpers for MySQL, NATS, Redis, MinIO, ClickHouse, Trino, and k3s. More involved tests can start an entire data-lake stack consisting of MinIO, an Iceberg REST catalog, and Trino.

Production-like configuration

The container should behave like the system our connector will encounter. As an example a Postgres instance with logical replication enabled to test CDC ingestion.

containerOpts := []testcontainers.ContainerCustomizer{
	postgres.WithDatabase(cfg.database),
	postgres.WithUsername(cfg.username),
	postgres.WithPassword(cfg.password),
	postgres.WithSQLDriver("pgx"),
	postgres.BasicWaitStrategies(),
}

if cfg.logical {
	containerOpts = append(containerOpts, testcontainers.WithCmd(
		"postgres",
		"-c", "fsync=off",
		"-c", "wal_level=logical",
		"-c", "max_replication_slots=10",
		"-c", "max_wal_senders=10",
	))
}

ctr, err := postgres.Run(ctx, cfg.image, containerOpts...)
if err != nil {
	t.Fatalf("start postgres container: %v", err

containerOpts := []testcontainers.ContainerCustomizer{
	postgres.WithDatabase(cfg.database),
	postgres.WithUsername(cfg.username),
	postgres.WithPassword(cfg.password),
	postgres.WithSQLDriver("pgx"),
	postgres.BasicWaitStrategies(),
}

if cfg.logical {
	containerOpts = append(containerOpts, testcontainers.WithCmd(
		"postgres",
		"-c", "fsync=off",
		"-c", "wal_level=logical",
		"-c", "max_replication_slots=10",
		"-c", "max_wal_senders=10",
	))
}

ctr, err := postgres.Run(ctx, cfg.image, containerOpts...)
if err != nil {
	t.Fatalf("start postgres container: %v", err

containerOpts := []testcontainers.ContainerCustomizer{
	postgres.WithDatabase(cfg.database),
	postgres.WithUsername(cfg.username),
	postgres.WithPassword(cfg.password),
	postgres.WithSQLDriver("pgx"),
	postgres.BasicWaitStrategies(),
}

if cfg.logical {
	containerOpts = append(containerOpts, testcontainers.WithCmd(
		"postgres",
		"-c", "fsync=off",
		"-c", "wal_level=logical",
		"-c", "max_replication_slots=10",
		"-c", "max_wal_senders=10",
	))
}

ctr, err := postgres.Run(ctx, cfg.image, containerOpts...)
if err != nil {
	t.Fatalf("start postgres container: %v", err

Likewise, our MySQL container enables row-based binary logging and GTIDs. These are not cosmetic settings. They determine whether the tests exercise the same replication protocols and checkpoint formats used by the real connector.

Manufacturing chaos

A real database is necessary, but it is not the test. The test still has to force the timing, state transitions, and database specific behavior that cause production failures.

One of the hardest problems in a data movement system is recovery. What happens when a run fails after some shards have completed but before the entire snapshot has landed? What happens if the source changes before the run resumes?

We built a recovery matrix that crosses our resumable PostgreSQL read strategies with operations that can occur during that failure window.

The read strategies currently include keyset scans, bitmap scans, and physical ctid scans using an xmin horizon. The concurrent operations include inserts, deletes, and updates, VACUUM FREEZE, VACUUM FULL, and transactions containing savepoints.

The test starts a real PostgreSQL database, seeds several tables, and records the primary keys present at the beginning of the run. It then wraps the real PostgreSQL sink with a deliberately unreliable sink that fails after a few writes:

sinks.Register("postgres_typed", func() filament.Sink {
	sink := pgsink.New()

	if firstRun.CompareAndSwap(true, false) {
		return testutil.NewFlakySink(sink, 5)
	}

	return sink

sinks.Register("postgres_typed", func() filament.Sink {
	sink := pgsink.New()

	if firstRun.CompareAndSwap(true, false) {
		return testutil.NewFlakySink(sink, 5)
	}

	return sink

sinks.Register("postgres_typed", func() filament.Sink {
	sink := pgsink.New()

	if firstRun.CompareAndSwap(true, false) {
		return testutil.NewFlakySink(sink, 5)
	}

	return sink

This is an important distinction: we mock the failure we want to inject, but we do not mock the database behavior we want to observe.

Once the run enters a resumable state, the test modifies the source database and requests the same run again. The runner loads its checkpoints, skips completed work, and continues from where it left off.

Filament provides at-least-once delivery, and our primary-key upsert sinks are idempotent. Re-reading a row is safe. Skipping one is data loss.

More than Postgres

We apply the same principle to our other connectors.

Our MySQL integration tests exercise typed snapshots, parallel keyset extraction, incremental backfills, and CDC operation ordering. The fixtures include unsigned integers near their maximum value, binary data, high-precision decimals, microsecond timestamps, JSON, and null values.

The object store sink tests use MinIO to exercise the complete object lifecycle. An object must not become visible as successful before commit, an aborted multipart upload must not leak parts, and an aborted run must not publish its success marker.

These tests are narrower than a complete workflow. They call a connector or service directly, but the dependency on the other side is real. That is the line we use between integration and end-to-end tests.

Crossing the process boundary

Our heaviest pipeline test compiles the Filament API server and control plane into separate binaries. They communicate through HTTP, PostgreSQL, and NATS rather than through in-process function calls.

The test starts a persistence database, a source database, a destination database, and a NATS server. It runs the database migrations, starts both services, and waits for their health endpoints.

Then it deliberately stops the control plane before submitting a pipeline run through the API.

The run must remain queued in NATS while there is no consumer available to execute it. After the control plane restarts, its durable subscription must receive the request, run the pipeline, and persist the final state.

The test does this twice. Between runs, it updates one source row, deletes another, and inserts a third. The destination is compared against the exact expected values after each restart.

This catches problems in configuration, migrations, serialization, durable messaging, process startup, and the HTTP boundary that an in-process test cannot cover.

Making real infrastructure practical

The trade-off is obvious, these tests are heavier than unit tests. They pull images, start processes, create databases, and wait for services to become ready.

Where possible, tests within a package share an infrastructure container. Shared state is reset between tests rather than creating a new service every time.

Integration and end-to-end tests live behind separate Go build tags. Ordinary unit tests remain fast and Docker-free, while CI gives each heavier tier its own job and timeout.

And of course… we still write unit tests.

Beyond mocking

The passing unit-tests meme is funny because most of us have experienced it. Every isolated component is green, but the system falls apart as soon as the components have to interact.

For Filament, those interactions are the product. Data has to cross databases, queues, processes, object stores, catalogs, and query engines without being lost or silently changed.