A Look Inside of Filament

Earlier this year we released Filament, our open-source data movement engine. We have been using Filament (or some variation of it) internally for a while now so we were extremely excited to release it publicly. The initial feedback we’ve gotten has been very rewarding for myself and the team! Now that the dust has settled a bit post release we decided now is a good time to dive a bit deeper into how Filament works, and what actually happens inside the pipeline. In this blog I will walk you through a single database table replication through the Filament engine. How are runs configured or started? How do we handle checkpointing? What if an error happens? What parts of the architecture are responsible for what actions? These are important questions to consider when choosing a data movement engine, even more so if you are self hosting. The hope is by the end of this blog you’ll have a much richer understanding of Filament and the systems that make it work.

What happens inside of Filament?

Say we’re copying an orders table into another system. The source has plenty of rows to send, the destination takes a moment to write each batch, and halfway through the run something goes wrong. Where does the next attempt begin?

That question is a good way to dive into Filament’s engine. It has to keep data moving without holding the whole table in memory. It also has to remember progress without getting ahead of what the destination has actually kept. Let’s walk through one row and you’ll see how the pieces fit together.

Creating a run

Before we can understand the pipeline we need to understand the architecture it runs in. There are a few different services with clear boundaries and responsibilities.

The Server receives RPCs requesting a run to execute. It creates a run in the database, then sends a run.requested event to the bus.

The Control Plane picks it up and dispatches a worker. If Filament is deployed in a Kubernetes environment, that usually means creating a Job dedicated to this run.

The Worker loads the saved request, then announces that it has started to work. From here on, it can focus on the data transfer and ignore most other parts of the system.

The Tracker, a component of the Control Plane, listens for events from the worker and records what happened. (If you open the UI while the copy is running, you can see the live event bus feed.)

These all come together to initialize a run in Filament.

Confirming the plan

Once the Worker has started the specified run, it needs to turn the pipeline definition into a concrete plan. It resolves the source’s credentials, configures the connection, and works out which resources, columns, and partitions are involved. For our example, it ends up with the orders table and the fields selected to be moved.

The Worker then checks how this resource should move. A full read starts at the beginning. An incremental read starts from a saved cursor. A stream follows ordered changes over a sequence. Each way of reading affects what progress means and how it is confirmed to be written. On the destination side, the sink advertises which kinds of writes it can perform (Append, Upsert, Deletes). The sink must also report when writes are durable enough to support a valid checkpoint. With S3, for example, writing a single part of a multi-part file doesn’t mean the records can be considered complete, because completing the upload could still fail. Filament uses all of this information and brings those capabilities together into a policy for the pipeline to execute from.

The destination is prepared before the first row arrives. If it needs a table or schema, this is when the runner gets that ready. It opens a destination session and begins the write. The source can start feeding rows once that session is in place.

Defining a batch

The source reads the first order. It hands the values to a row writer in the agreed schema order, along with a source position when that read has one. Another order follows, then another. Filament collects them in typed Arrow columns, so it can move a group of rows through the engine as one batch.

The group has a few thresholds to decide when it’s complete. It closes when it reaches the configured number of rows or an approximate byte size. A timer can ask for a partial batch to move along when rows arrive slowly; the builder honors that request after the next complete row. The source can flush explicitly as well. Once a batch is ready, the pipeline takes ownership of it and the source starts filling the next one.

Now suppose the destination slows down for any number of reasons. Ready batches collect in a bounded queue of configurable size. When that queue fills, the source waits while handing off the next batch. As soon as a writer makes room, extraction continues. The rate of reading settles to the rate the destination can sustain, without a growing backlog of rows in memory.

A transform can also run over an entire Arrow batch. When one is configured for orders, it runs on batches between the builder and the destination writer in a separate transform queue. It changes the batch that will be written. This does mean the integrity check needs to account for the transformed batch, so when it checks the data the destination received we know the transformed modifications persisted.

Writing with confidence

A writer takes the first batch from the queue and hands it to the destination. The destination applies it according to the policy chosen during the planning stage, then returns a receipt. This receipt contains useful data like how many rows and bytes it handled, a checksum, and sometimes a URI or checkpoint depending on the destination.

Filament also calculates a checksum from the batch it sent. It compares that value with the destination’s checksum from the write. When they match, the pipeline emits batch.written and integrity.verified. If they differ, it reports the divergence and stops the run.

A commit receipt is useful evidence for this batch. The runner waits for the source to finish and for every queued batch to leave the pipeline. Then it closes that session with Commit. If a write fails, it records the failure and handles the unfinished session according to whether the run can resume.

What if something goes wrong?

Let’s say the source has read through order 200. It can report a cursor at that position, and the last batch can carry it forward. But the cursor tells us where reading got to. It doesn’t, by itself, tell us what a future run can skip.

The tracker listens for batch.written events and uses them to build a picture of the progress. It adds the row and byte counts to the run and combines checkpoint information for each resource. A source may also report that its watermark has advanced; the tracker waits for the batches carrying that progress to be written before using it as checkpoint input.

Now put a failure between the last batch and the commit. The source may have read through order 200, but the destination has not confirmed it was written. The cross-run checkpoint stays at its earlier safe position. The next run may read some orders again. This can lead to extra reading, however it’s how Filament avoids jumping past data whose final write outcome was never confirmed.

A deliberate pause follows similar rules. The runner stops new extraction, lets batches already accepted by the pipeline drain, and reaches a commit or abort boundary that fits the planned policy.

Events hold the full journey of a row

As all this happens, the engine sends small, typed events about what it has seen and done: a resource started, a batch was buffered, a batch was written, an integrity check passed, a run finished. The event bus carries those events to the tracker, which is responsible for updating state across the system.

Events can occasionally be delivered more than once, however the tracker uses the event bus sequence to recognize a delivery it has already folded, so an orders batch does not count twice. At the end of a run it flushes the relevant checkpoint information before storing the final status. If a worker has vanished, a failure event from the reaper follows that same path into run state.

The pieces each own a clear part of the journey. The connector reads the row. The pipeline collects it, applies any transform, and checks the destination’s receipt. The runner opens and closes the destination session. The tracker is responsible for consuming all this data. Together they make up the Filament engine.

And that’s it! Well… sort of. This blog is more about peeling back the curtain than an in-depth explanation of every part of the system. Data movement is full of never ending nuance, caveats, and configuration as connectors and features expand. Maybe it’s best to view this blog as the first of many to come. If there is any part of the system or a specific connector you as a reader or user want more insight into, whether that’s the decisions we made or how it works, please reach out! We love talking about our work and even more so love to improve it. If you are moving data (with Filament or not) we would love to know what you’re moving and why, and especially any pain points you’re experiencing.