Best Way to Sync GitHub to ClickHouse in 2026

If your engineering dashboards live in ClickHouse and your code lives in a GitHub organization, this guide covers every tool that joins the two.

Four paths land GitHub data in ClickHouse as of November 2026. A self-run engine such as Filament, dlt, or ingestr reads the REST API and writes typed MergeTree tables on your compute. Managed ELT such as Airbyte Cloud, Fivetran, or Estuary runs the sync for a fee metered in credits, rows, or gigabytes. GH Archive covers the public timeline only, and webhooks alone see deletes.

How the GitHub API shapes every ClickHouse sync

Every tool on this route shares one request budget and one quirk. The rate limit docs give a token 5,000 requests an hour, and the pagination docs cap most pages at 100 rows.

The first row is the one that breaks dashboards. In GitHub’s own words, “GitHub’s REST API considers every pull request an issue, but not every issue is a pull request.” Land the issues resource without that marker and every pull request counts twice.

Five criteria for picking a GitHub to ClickHouse tool

Five criteria decide this route. Throughput is not one of them, because the API budget caps every tool at the same ceiling.

• Incremental cursors. The tool should pass since on issues and comments instead of re-reading every repository on every run, because those are the only cheap endpoints.

• Delete and churn handling. Stars, labels, and team membership churn without a delete signal, so those tables need a scheduled full replace or a keyed merge.

• Engine and dedup. ClickHouse upserts land in ReplacingMergeTree, and the engine docs state that “Data deduplication occurs only during a merge.” A tool should either stage and swap atomically or tell you to query with FINAL.

• Shape in ClickHouse. Nested GitHub JSON needs typed columns for the fields you aggregate plus a raw copy of the rest. The JSON type is production ready since 25.3, yet most tools still write String.

• The unit you pay by. Credits, monthly active rows, gigabytes, a CLI token, or your own compute, and nightly pull request re-reads across hundreds of repositories punish a row meter most.

Which tools can sync GitHub to ClickHouse?

Nine tools name GitHub as a source and ClickHouse as a destination, and the split is who runs the process and what gets metered.

Airbyte has the longest stream list and the sharpest warning. Its source docs say only comments, commits, issues, and review comments are pure incremental, so the rest spend the API budget like a full refresh. The certified ClickHouse destination writes typed columns to ReplacingMergeTree and drops a column when the source drops it. Cloud starts at 20 dollars a month with hourly syncs.

Fivetran is the one tool here that hard deletes GitHub rows, on join tables such as issue labels and team memberships, per its connector page. Its partner-built ClickHouse Cloud destination went generally available in May 2026 and does not list self-hosted ClickHouse. The free plan stops at 500,000 monthly active rows.

Estuary’s GitHub capture is an adapted Airbyte connector, and its ClickHouse materialization writes over the native protocol. It bills 0.50 dollars per gigabyte plus 100 dollars a month per connector above a free tier, per its pricing, under the Business Source License.

The code-first tools are cheapest and narrowest. dlt’s verified source covers one repository, while its ClickHouse destination swaps staging tables in with EXCHANGE TABLES. ingestr reloads issues in full every run, per its source page, under an FSL license. Sling’s GitHub connection needs a CLI Pro token, and CloudQuery’s ClickHouse plugin is append only.

Five familiar names cannot run this route. ClickPipes has no REST source, Hevo and Stitch have no ClickHouse destination, and PeerDB and Debezium read databases, not APIs.

Where Filament fits on GitHub to ClickHouse

Filament reads one organization through a YAML manifest and writes typed MergeTree tables. Against the five criteria it scores on cursors, engine handling, shape, and cost, and it handles churn with a scheduled replace. The GitHub source and the ClickHouse sink are both listed out of preview in the connector overview. The rule for that flag is to “Clear the flag after live validation confirms the supported behavior.”

Disclosure, this site belongs to Galaxy, the company behind Filament.

On cursors, issues, issue comments, and review comments read incrementally through since with one durable watermark per resource and a 300 second re-read window. Every other resource is a full read that resumes after its last completed page if interrupted.

On the engine, the sink page builds replace runs in a staging table and swaps them in at commit with EXCHANGE TABLES, so the old table stays queryable until the new one is ready. Upserts go to a live ReplacingMergeTree on the primary key, and a CRC32-C checksum covers every batch before the write, per the integrity guide.

On cost there is no meter, because the engine is Apache 2.0 and runs as one binary on a laptop or on Kubernetes through the Helm chart, so the bill is your compute plus ClickHouse.

Four limits are stated on the docs pages. Change data capture, the log-based mode that carries deletes, is off the table. The sink page says that “CDC and incremental delete are not supported because ClickHouse does not offer efficient row-level deletes for this sink.” Schema evolution is add-only, the source takes a single token at one request per second, and the manifest targets github.com with no Enterprise Server base URL.

The public benchmark cohort has run no GitHub or ClickHouse route, and a route without a public result file gets no throughput claim on this page.

How to sync GitHub to ClickHouse with Filament

One fine-grained token, one ClickHouse user, and three pipelines cover an organization. The sink block is copied from the Postgres to ClickHouse how-to. The source flags apply the CLI guide’s field-to-flag rule to the connector’s two config keys, because the docs carry no GitHub CLI example yet.

Install the CLI with brew install galaxy-io/tap/filament or the install script, then create the source and discover it.

filament source create acme-github \
  --source-connector github \
  --source-token-env GITHUB_TOKEN \
  --source-organization acme

filament source

filament source create acme-github \
  --source-connector github \
  --source-token-env GITHUB_TOKEN \
  --source-organization acme

filament source

filament source create acme-github \
  --source-connector github \
  --source-token-env GITHUB_TOKEN \
  --source-organization acme

filament source

Create the sink. It defaults to the secure native port 9440, which ClickHouse Cloud expects, and a self-hosted server without TLS adds --sink-port 9000 --sink-secure=false. The user needs access to the default database, where the sink runs CREATE DATABASE.

filament sink create eng-clickhouse \
  --sink-connector clickhouse \
  --sink-host abc123.us-east-1.aws.clickhouse.cloud \
  --sink-password "$CLICKHOUSE_PASSWORD"

filament sink create eng-clickhouse \
  --sink-connector clickhouse \
  --sink-host abc123.us-east-1.aws.clickhouse.cloud \
  --sink-password "$CLICKHOUSE_PASSWORD"

filament sink create eng-clickhouse \
  --sink-connector clickhouse \
  --sink-host abc123.us-east-1.aws.clickhouse.cloud \
  --sink-password "$CLICKHOUSE_PASSWORD"

Create three pipelines, the split the source page recommends. The catalog pipeline replaces membership tables that churn, and the activity pipeline upserts the three incremental resources. The traffic pipeline upserts views and clones so history outlives GitHub’s 14 day window.

filament pipeline create github-catalog \
  --source acme-github \
  --sink eng-clickhouse \
  --resources repositories,repository_details,stargazers,forks,labels,milestones,contributors,languages \
  --sync-mode full \
  --write-mode replace \
  --sink-database github

filament pipeline create github-activity \
  --source acme-github \
  --sink eng-clickhouse \
  --resources issues,issue_comments,pull_request_review_comments \
  --sync-mode incremental \
  --write-mode upsert \
  --sink-database github

filament pipeline create github-traffic \
  --source acme-github \
  --sink eng-clickhouse \
  --resources traffic_views,traffic_clones \
  --sync-mode full \
  --write-mode upsert \
  --sink-database

filament pipeline create github-catalog \
  --source acme-github \
  --sink eng-clickhouse \
  --resources repositories,repository_details,stargazers,forks,labels,milestones,contributors,languages \
  --sync-mode full \
  --write-mode replace \
  --sink-database github

filament pipeline create github-activity \
  --source acme-github \
  --sink eng-clickhouse \
  --resources issues,issue_comments,pull_request_review_comments \
  --sync-mode incremental \
  --write-mode upsert \
  --sink-database github

filament pipeline create github-traffic \
  --source acme-github \
  --sink eng-clickhouse \
  --resources traffic_views,traffic_clones \
  --sync-mode full \
  --write-mode upsert \
  --sink-database

filament pipeline create github-catalog \
  --source acme-github \
  --sink eng-clickhouse \
  --resources repositories,repository_details,stargazers,forks,labels,milestones,contributors,languages \
  --sync-mode full \
  --write-mode replace \
  --sink-database github

filament pipeline create github-activity \
  --source acme-github \
  --sink eng-clickhouse \
  --resources issues,issue_comments,pull_request_review_comments \
  --sync-mode incremental \
  --write-mode upsert \
  --sink-database github

filament pipeline create github-traffic \
  --source acme-github \
  --sink eng-clickhouse \
  --resources traffic_views,traffic_clones \
  --sync-mode full \
  --write-mode upsert \
  --sink-database

Verify with FINAL, because upserted parts merge later. The second query is the double count check.

SELECT count() FROM github.issues FINAL;

SELECT repository,
       countIf(pull_request IS NULL) AS issues,
       countIf(pull_request IS NOT NULL) AS pull_requests
FROM github.issues FINAL
GROUP BY repository ORDER BY

SELECT count() FROM github.issues FINAL;

SELECT repository,
       countIf(pull_request IS NULL) AS issues,
       countIf(pull_request IS NOT NULL) AS pull_requests
FROM github.issues FINAL
GROUP BY repository ORDER BY

SELECT count() FROM github.issues FINAL;

SELECT repository,
       countIf(pull_request IS NULL) AS issues,
       countIf(pull_request IS NOT NULL) AS pull_requests
FROM github.issues FINAL
GROUP BY repository ORDER BY

Every row also carries _filament_run_id and _filament_run_started_at, so a dashboard can exclude a failed run. Schedule the activity pipeline every 15 minutes and the catalog pipeline nightly in the web app or the Pipelines API. Timestamps land as DateTime64 in UTC, and user, labels, and raw land as String.

Matching the tool to the engineering question

Organization scope and tolerance for a meter decide the pick, and a self-run engine takes the common case of one organization in typed tables with no vendor bill.

Fivetran wins when deleted labels and memberships must vanish without a replace, on ClickHouse Cloud only. Airbyte wins on stream breadth and token rotation for very large organizations, and GH Archive wins when the question is about GitHub rather than your GitHub.

Related guides

GitHub to Snowflake and GitHub to Postgres move the same source into other sinks. Postgres to ClickHouse and MySQL to ClickHouse cover the database routes into this sink. The write modes explainer defines replace and upsert, the REST API manifest guide shows how a SaaS source is declared, and Introducing Filament carries the launch context.

Frequently asked questions

Does ClickHouse have a native GitHub connector?

No. ClickPipes, the managed ingestion service in ClickHouse Cloud, reads Kafka, Kinesis, Pub/Sub, object storage, and database change streams, and it has no REST or SaaS source. Something else has to call the GitHub API and insert rows, whether a self-run engine, a managed ELT service, a library, or a script.

Why does my GitHub issues table in ClickHouse contain pull requests?

Because the GitHub REST API treats every pull request as an issue. The issues endpoint returns both, and each pull request row carries a pull_request key. Filter on that key before counting open issues, or every pull request counts twice. Most connectors keep the marker as a column, so the filter is one WHERE clause.

Why do I see duplicate rows after syncing GitHub into ClickHouse?

Most tools write upserts to a ReplacingMergeTree table, and that engine removes duplicates only when parts merge in the background. Until then a key can have several rows. Add FINAL to the query to read the current state, or run OPTIMIZE TABLE with FINAL to force a merge.

How do deleted GitHub issues and removed stars get out of ClickHouse?

Only through a full replace or a webhook. The REST list endpoints never return a deleted issue, comment, or removed star, so an incremental sync on updated_at cannot see the removal. Rebuild membership tables such as stargazers with a scheduled full replace, or receive the GitHub webhook deleted actions and apply them yourself.

What is the difference between loading GH Archive and syncing your own GitHub organization?

GH Archive is the public event timeline, aggregated into hourly JSON files since February 2011, and ClickHouse publishes it as a dataset of billions of rows. It holds no private repositories, traffic counts, members, or teams. An organization sync reads your own repositories through the API with a token, private ones included.

Which GitHub permissions does a read-only sync token need?

A fine-grained token needs Metadata read for repositories, Issues and Pull requests read for issues and comments, Members read on the organization for teams, and Administration read for traffic. Since July 2026 stargazer listings need a token from a repository admin or collaborator, or the list comes back empty.

How does Filament write GitHub data into ClickHouse?

A YAML manifest reads one organization through the REST API, and the ClickHouse sink writes each resource to its own typed table. Replace mode builds a staging table and swaps it in atomically at commit. Upsert mode writes to a live ReplacingMergeTree, so issues and comments arrive incrementally through since and merge by primary key.

How does Filament stay inside the GitHub rate limit?

The GitHub manifest throttles to one request per second, which leaves room inside one token’s hourly budget of 5,000 requests, and it honors Retry-After and the reset header whenever GitHub throttles. It uses a single token with no documented rotation, so a very large organization takes longer rather than failing.