Introducing Filament

Today, we are open sourcing Filament.

Filament is a data movement engine built by data engineers, for data engineers, that carries information from wherever your data lives to wherever you need it, quickly and correctly, with the care of something meant to last and to hold up at enormous scale. It is the start of our love letter to the data community.

Every company today has more knowledge than it has ever had, spread across databases and SaaS tools and object storage that were never designed to talk to each other, and yet they need to for any of it to become truly useful. When data flows reliably between those systems, teams stop fighting their tools and start acting on what they know, whether that means understanding their customers, running their operations, or putting AI to work on problems that actually matter. We have spent most of our careers on exactly that, and we want to support a greater ecosystem of powerful teams and companies that want to do great things. That starts with infrastructure they can own, so we built the engine we always wished existed, and we are sharing it with the world.

Filament is Apache 2.0 licensed, runs anywhere, and takes about a minute to get going. This post is the story of why it exists, what we believe, and where we are taking it. If you just want to get started, the docs and the repo are here.

A little history

Galaxy was started by the two of us, Mitch Bregman and Leon Kozlowski. We met in college pledging the same fraternity, became roommates, and somehow never stopped building things together. Years later that landed us on the same team at Flock Safety, early, when the company was small and growing fast and generating an enormous amount of data. We were the people responsible for making that data useful. Every good thing we wanted to build sat downstream of one unglamorous problem. Get the data in. Get it in correctly. Get it in fast enough that the analytics, the search, and everything else on top of it ran smoothly.

So we did what everyone does. We bought a tool. A well known managed ETL vendor that charged by the row. It worked, for a while. Then our volume kept climbing, and one day they told us we were done. We were moving so much data that they were losing money on us, and they asked us to leave.

We laugh about it now. At the time it was a real problem. A system our company depended on had just decided that our growth was inconvenient. So we built our own. Extraction, batching, retries, checkpoints, schema handling, all of it. It was a lot of work, and it ran better than anything we had paid for.

That experience never left us. Moving data is the layer everything else depends on, and the tools for it were slow, expensive, or priced in a way that punished you at exactly the moment you needed them most.

We want to be fair to the giants here. Fivetran, Airbyte, and dbt made data movement and modeling a thing that whole teams organize around, and they made it good enough that people stopped writing everything by hand. Luigi, the orchestrator out of Spotify, is where we built our first real pipelines and is where our love for this work started. We would not be here without them, and Filament stands on everything they taught the industry.

How Galaxy got here

Galaxy started as a query editor. We wanted a fast, modern SQL workspace, so we built one as a native desktop app, and people liked it. We built it on Wails, the open-source framework for writing desktop apps in Go, and along the way we met Atterpac, one of its maintainers. He is an exceptional engineer, exactly the kind open source runs on, and luckily we were able to convince him to join the team. He has spent years building something a whole community adopted and loves, and he has been generous enough to share everything he learned along the way with us. So much of how we design Filament and how we think about building in the open comes from him, and we are grateful every day that he said yes.

The editor taught us something. Writing the query was never the hard part. The hard part was understanding the question someone was really asking, knowing where the information lived to answer it, and answering it in a way people could trust. Which table was the real one. Which definition of "customer" the finance team used. Who decided what status_v2 meant. All of that lived in people's heads and scattered docs, and beyond the answer itself was the harder question of what people actually did next, how they reacted and acted on what they had just learned. Every AI feature we shipped ran straight into that wall.

We fell in love with that problem, understanding what a company's information means, how to parse it, how to connect it, and how to make it usable by people and by AI. That is what a context management system is. Some people call it a context layer, some call it a company brain, and we think it is the natural evolution of the data platform in the era of AI. The warehouse held your data. The context management system holds your data, your definitions, and your institutional knowledge together, structured and governed, so anything you build on top of it starts out already understanding your business. That is the problem we are solving at Galaxy.

The very first thing that system needed was a way to bring data in. We started with open-source tools and wrote custom pieces around them. They were slow and heavy, and getting them deployed was its own project. Sidecars, orchestrators, a maze of containers, and important features tucked behind a paid tier. We did not want to fight the tool. We wanted something we could stand up in an afternoon, run anywhere, and understand end to end.

We also needed more than "get the rows over there." Before you can model anything, you have to move it, and you have to control every aspect of how it moves. What types come through. What gets replaced, what gets appended, what gets merged. What happens when a run fails halfway. You need fine grained control over what enters the system, because everything you build on top inherits whatever sloppiness you let in.

So we built the engine we needed, inside our context management product, and it worked. Then we made a decision. We pulled it out, gave it a name, and made it open source. That engine is Filament.

What we believe

Our mission is to build world-class open source data infrastructure. The goal is to build the greatest context management system, and to build it in the open, piece by piece, so that anyone can run it, inspect it, extend it, and own it. Here is what sits underneath that.

Every AI deployment runs on a context layer, regardless of what you call it. When a model answers a question about your business, it is drawing on some collection of your data, your definitions, and your history. If that collection is fragmented or stale, the answer is fragmented or stale. The companies getting real value from AI are the ones treating context as infrastructure and building it on purpose.

Data movement is step one. You cannot govern what you have not gathered. You cannot structure data that is still scattered across a dozen SaaS tools and a handful of databases. The context layer starts with an engine that can pull from anywhere, land anywhere, and keep everything current. If that engine is slow, everything above it is slow. If it is sloppy, everything above it is untrustworthy.

Your knowledge should be yours. It is one of the most valuable things a company has, and the infrastructure that holds it should not be rented, metered, or locked in a format that costs money to leave. That is why Filament is Apache 2.0.

Pricing by the row is backwards. It charges you the most when your business is working the best. It makes engineers ration data, sync less, and leave things out. The layer underneath your context should want you to move more data, not less.

Galaxy is our love letter to the data community. We are data engineers. We have been on call for the pipeline at three in the morning. We have explained to a finance team why the numbers moved. We know what this work feels like, and we want to make it better for the people doing it.

What Filament does

Filament is a data replication engine. Here is what that means in plain terms.

It reads from your sources and writes to your destinations. Databases like Postgres and MySQL. Warehouses and lakes like ClickHouse and Apache Iceberg. Object storage like S3. SaaS tools like Stripe, GitHub, Notion, Linear, Slack, and PostHog. You connect a source to a sink, pick the tables or objects you care about, and run it.

It moves data the way you tell it to. Copy everything fresh. Pull only what changed since last time. Or listen to the database's change log and stream inserts, updates, and deletes as they happen. On the other side, replace, append, upsert, or merge. Filament checks that the combination makes sense before it moves a single row.

It checks its own work. Every batch is verified as it is written. If anything changed in flight, the run fails loudly instead of quietly landing bad data.

It picks up where it left off. Progress is saved as work is confirmed, so a run that dies halfway resumes from the last safe point instead of starting over.

It is easy to extend. Adding a new SaaS source usually means writing a short YAML file, not a new codebase. The repo even ships a skill so a coding agent can research an API and write the connector for you.

It runs wherever you do. One command gives you a standalone binary with a web UI for trying it out. Ten lines of Go embed it inside your own service. A Helm chart runs it on Kubernetes with isolated workers and full observability. Same engine, no gates, nothing held back for a paid tier.

docker run -p 8080:8080 ghcr.io/galaxy-io/filament/standalone

docker run -p 8080:8080 ghcr.io/galaxy-io/filament/standalone

docker run -p 8080:8080 ghcr.io/galaxy-io/filament/standalone

Filament is pre-1.0. Some connectors are early and the interfaces may still shift. The docs are honest about where it stands.

It is also fast

We are proud of this part, so we will keep it short and let the numbers speak elsewhere.

We built an open benchmark and ran Filament against the tools people actually use, on dedicated cloud databases, with every result validated. On the biggest test, a year of NYC taxi trips from Postgres to Postgres, Filament moved 298 million rows in under two minutes. That is more than four times faster than piping pg_dump into psql, and the other tools we tested took anywhere from a few minutes to almost three hours to move the same rows. Filament also used a fraction of the memory and CPU to do it.

Filament outperformed every other tool in five of the six scenarios we ran. In the sixth, one tool finished 13 percent sooner while spending fifteen times the CPU to get there. We published every number, every caveat, and the harness itself, so you can rerun it and check our math. The full benchmark report has all of it.

What comes next

Filament is the first open-source component of the context management system we are building. It will not be the last.

AI is a data problem. Every impressive demo and every disappointing rollout comes down to the same question. Did the model have the right context, and could it trust it? The rest of our stack is our answer, and it is where we have spent the past year. Semantic modeling. Entity resolution across systems that disagree about who a customer is. Ontologies that hold your definitions in a form both people and models can read. Governance and access control so the right context reaches the right agent and nothing else does.

And then, action. A data platform that only answers questions is half a platform. The point of understanding your business this well is to do something with it. Kick off the workflow. Update the record. Flag the account. Draft the reply. Once a system understands what your data means, it can act on what it has learned, and it can do so with guardrails, so the actions are consistent, auditable, and safe. That is where we are taking this, and we are going to keep shipping the pieces in the open as we go.

Filament is what we wished someone had handed us years ago. So we are handing it to you.

Come build with us

If you want to deploy Filament inside your company, we will help you do it. If you want to partner with us, whether you are a warehouse, a platform, or a team building your own context layer, we would love that. Reach out and we will roll up our sleeves with you.

And if you just want to kick the tires, go run it. Break it. Tell us what is missing. Open an issue, send a pull request, or come hang out in the Filament Slack.

Welcome to Filament.

Mitch, Leon, and the Galaxy Team