Best Way to Sync GitHub to Postgres (2026)

This guide is for an engineering lead who wants pull request, commit, and CI data in Postgres for DORA-style metrics without paying per row. As of October 2026 the job has six shapes, because GitHub ships no Postgres export of its own. The choice is managed ELT, a self-hosted engine such as CloudQuery, Meltano, or Filament, a foreign data wrapper, webhooks, a row automation, or a script.

What the GitHub API gives a sync tool

GitHub meters requests per hour, filters issues and commits by update time but not pull requests, and never lists deletions, and those three facts set the shape of every tool below. The figures sit in the table, each from GitHub's REST docs.

The rate limit is the constraint that decides architecture. GitHub documents the hourly figure, the points per minute, and the retry header, and adds that "There is not a way to check the status of your secondary rate limit" (GitHub). A tool that fans out across repositories in parallel finds the ceiling by hitting it.

The missing update filter on pull requests is the second trap. Issues accept since and sort by updated (GitHub), but the pull requests endpoint offers only sort and direction (GitHub). Every tool therefore re-reads PRs each run or leans on the issues list, which returns pull requests too with a pull_request marker.

Retention is the third. Since the start of October 2026 the Actions retention setting governs "checks, workflow runs, and statuses" and "Changing the setting won't restore data that was previously removed" (GitHub changelog). Traffic views and clones cover only the last 14 days (GitHub).

What decides a GitHub to Postgres setup?

Five constraints decide the method, and the request budget eliminates more designs than the other four combined.

• Request budget, meaning how many repositories and open pull requests must be re-read each run inside 5,000 requests an hour, and whether the tool rotates tokens or honors Retry-After.

• Object coverage, meaning whether reviews, review comments, workflow runs, releases, and stargazers land as their own tables or only issues and pull requests.

• Mutation handling, meaning whether closed, merged, and deleted items are reflected through updated_at cursors, full replace, or webhook events.

• Table shape, meaning typed columns with a primary key per object versus a JSON document per API response.

• Billing unit and data path, meaning per monthly active row, per credit, per row synced, per gigabyte, or nothing, and whether the data leaves your network.

The request budget is arithmetic once you know that every page holds at most 100 items (GitHub). The table shows what a nightly run costs per repository.

How can you sync GitHub to Postgres?

Every tool on this route re-reads pull requests, so coverage, token handling, and the meter decide. The table lists each option that runs the route today, from each vendor's docs.

Price is the other axis and the units do not line up, so the table shows each managed tool's entry tier as of October 14, 2026. The self-hosted engines bill nothing, except CloudQuery, which meters rows synced per year.

Fivetran and Stitch are the managed defaults. Fivetran captures hard deletes on four tables and recently added a default-branch-only commits option, but its setup guide asks for four or five tokens from different users to stay inside the rate limit (Fivetran). Stitch runs the Singer tap, bills by rows, and has no workflow run table (Stitch).

Airbyte has the widest stream list and the most candid incremental disclosure. Its docs say only comments, commits, issues, and review comments "are pure incremental" and the other 19 incremental streams read everything and emit only new records (Airbyte). Estuary ships the same stream set under BSL 1.1 and adds a real-time mode (Estuary).

CloudQuery and Meltano are the self-hosted tools built for this exact job. CloudQuery syncs issues incrementally on updated_at, writes to Postgres with COPY and ON CONFLICT, and recommends GitHub App auth for the higher limit, but it is priced on rows synced per year (CloudQuery). Meltano's tap-github rotates through additional_auth_tokens when it hits the limit and is Apache 2.0 (Meltano).

The remaining paths are narrower. dlt and ingestr read issues and pull requests in replace mode and only the 30-day events feed incrementally (dlt). Hevo connects to GitHub "through Webhooks" with no API backfill (Hevo). Steampipe installs as a native Postgres foreign data wrapper for ad hoc joins (Steampipe), and Zapier or Make insert a row per trigger with no history.

Where Filament fits on GitHub to Postgres

Filament fits the org-wide nightly copy with typed tables and no meter, not the team that needs review events within seconds. Filament is built by Galaxy, which also publishes this site. The GitHub source carries a beta label, meaning it runs end to end but is not verified for every object or environment, and the Postgres sink carries the same label.

On the request budget, the manifest throttles to one request per second, which its own comment frames as room inside the standard hourly budget. It honors Retry-After and the budget-reset headers on throttled responses (Filament). There is no multi-token rotation, so a very large organization needs a GitHub App installation token for the higher ceiling.

On coverage, the source reads one organization's repositories and dozens of child resources, with a dozen selected by default and reviews, workflow runs, releases, security alerts, and Discussions opt-in. Pull requests, commits, and workflow runs are full reads every run, and commits are keyed by repository, branch, and sha, so one commit on three branches is three rows.

On mutations, issues, issue comments, and review comments read incrementally on updated_at. The docs say "Each resource shares a single durable watermark across repositories" and a 300-second overlap re-reads the window around each scan (Filament). Closed and merged pull requests land through the full read, and deleted items need a periodic full replace.

On table shape and the data path, the Postgres sink creates one typed table per resource with a primary key, writes with binary COPY, and merges with ON CONFLICT in one transaction (Filament). Evolution is add-only, and a raw jsonb column keeps unprojected fields. Nothing leaves your network, nothing is billed per row, and no speed figure appears here because no public run covers a GitHub route.

How to sync GitHub to Postgres with Filament

The route runs on a laptop with the local CLI, which uses file-backed SQLite and needs no Postgres or NATS of its own (installation). The GitHub source page shows no config example, so the YAML below is assembled from its two documented fields and the rules on the CLI page.

1. Create a fine-grained token and install the CLI

brew install galaxy-io/tap/filament
export GITHUB_TOKEN=github_pat_...
export METRICS_PG_DSN='postgres://filament:...@metrics.internal:5432/metrics?sslmode=require'

brew install galaxy-io/tap/filament
export GITHUB_TOKEN=github_pat_...
export METRICS_PG_DSN='postgres://filament:...@metrics.internal:5432/metrics?sslmode=require'

brew install galaxy-io/tap/filament
export GITHUB_TOKEN=github_pat_...
export METRICS_PG_DSN='postgres://filament:...@metrics.internal:5432/metrics?sslmode=require'

Grant the token read access to Metadata, Issues, Pull requests, Contents, and Actions, the permissions GitHub's docs map to these endpoints (GitHub).

2. Declare the source, the sink, and the nightly full pipeline

version: 1
sources:
  engineering:
    type: github
    config:
      token: env:GITHUB_TOKEN
      organization: acme
sinks:
  metrics:
    type: postgres
    config:
      connection_method: url
      dsn: env:METRICS_PG_DSN
pipelines:
  engineering-nightly:
    source:
      ref: engineering
    sink:
      ref: metrics
    resources:
      - repositories
      - pull_requests
      - pull_request_reviews
      - commits
      - workflow_runs
      - releases
    sync_mode: full
    write_mode

version: 1
sources:
  engineering:
    type: github
    config:
      token: env:GITHUB_TOKEN
      organization: acme
sinks:
  metrics:
    type: postgres
    config:
      connection_method: url
      dsn: env:METRICS_PG_DSN
pipelines:
  engineering-nightly:
    source:
      ref: engineering
    sink:
      ref: metrics
    resources:
      - repositories
      - pull_requests
      - pull_request_reviews
      - commits
      - workflow_runs
      - releases
    sync_mode: full
    write_mode

version: 1
sources:
  engineering:
    type: github
    config:
      token: env:GITHUB_TOKEN
      organization: acme
sinks:
  metrics:
    type: postgres
    config:
      connection_method: url
      dsn: env:METRICS_PG_DSN
pipelines:
  engineering-nightly:
    source:
      ref: engineering
    sink:
      ref: metrics
    resources:
      - repositories
      - pull_requests
      - pull_request_reviews
      - commits
      - workflow_runs
      - releases
    sync_mode: full
    write_mode

Save it as ~/.config/filament/filament.yaml. With no schema set, the sink names the schema after the source connection, here engineering.

3. Discover the resources and run the first load

filament source discover engineering
filament run engineering-nightly
filament run ls

filament source discover engineering
filament run engineering-nightly
filament run ls

filament source discover engineering
filament run engineering-nightly
filament run ls

An interrupted full read of a top-level resource resumes after its last completed page, so a backfill across hundreds of repositories survives a restart.

4. Add the hourly incremental pipeline for issues and comments

  engineering-hourly:
    source:
      ref: engineering
    sink:
      ref: metrics
    resources:
      - issues
      - issue_comments
      - pull_request_review_comments
    sync_mode: incremental
    write_mode

  engineering-hourly:
    source:
      ref: engineering
    sink:
      ref: metrics
    resources:
      - issues
      - issue_comments
      - pull_request_review_comments
    sync_mode: incremental
    write_mode

  engineering-hourly:
    source:
      ref: engineering
    sink:
      ref: metrics
    resources:
      - issues
      - issue_comments
      - pull_request_review_comments
    sync_mode: incremental
    write_mode

Each resource advances an <resource>_updated_at checkpoint that becomes durable only at a committed completion, so a failed run never skips rows (Filament).

5. Verify in Postgres and schedule both pipelines

SELECT repository_name, count(*) FROM engineering.pull_requests GROUP BY 1 ORDER BY 2 DESC LIMIT 10;
SELECT count(*) FROM

SELECT repository_name, count(*) FROM engineering.pull_requests GROUP BY 1 ORDER BY 2 DESC LIMIT 10;
SELECT count(*) FROM

SELECT repository_name, count(*) FROM engineering.pull_requests GROUP BY 1 ORDER BY 2 DESC LIMIT 10;
SELECT count(*) FROM

Column names follow GitHub's payload fields, so check the discovered schema before writing lead-time views. Add traffic_views and traffic_clones with full upsert if you want traffic history past 14 days.

Which GitHub to Postgres option fits which job?

A self-hosted engine is the pick for an org-wide typed copy with no row meter, and a managed tool is the pick when someone else should own the token rotation and the rate-limit alerts.

CloudQuery beats Filament when the job needs its security and Copilot tables and the team accepts per-row pricing. Meltano wins when token rotation across a very large organization matters more than typed tables.

Related guides

• REST API to Postgres, syncing any API with a YAML manifest

• Best Way to Sync GitHub to Snowflake (2026)

• Best Way to Sync Granola to Postgres (2026)

• Full Load vs Incremental vs CDC, which to use

• Best Postgres Replication Tools in 2026

• Filament GitHub source docs

Frequently asked questions

Does GitHub have a native export to Postgres?

No. GitHub's own export paths are audit log streaming to S3, Azure, Google Cloud Storage, Datadog, or Splunk on Enterprise Cloud, the Migrations API for repository archives, and the public GH Archive timeline. None writes issues, pull requests, or commits into Postgres, so the job needs a sync tool, a foreign data wrapper, or your own code.

How do I get GitHub issues and pull requests into a SQL database?

Point a sync tool at the organization with a fine-grained token that has read access to Metadata, Issues, Pull requests, Contents, and Actions. It pages the REST API at 100 items per request into typed tables. Fivetran, Airbyte, Stitch, Estuary, CloudQuery, Meltano, and Filament all do this, and Steampipe exposes the same data as Postgres foreign tables without a copy.

How many API requests does syncing a GitHub organization take?

A personal token gets 5,000 requests per hour and each page returns at most 100 items, so a full read costs roughly one request per page of issues, pull requests, or commits per repository. Conditional requests that come back unchanged are free. Pull requests have no since filter, so they are re-read every run, which is the budget line to watch.

Can a GitHub sync capture deleted issues or repositories?

Not from the list endpoints. The REST API has no delete endpoint for issues, a deleted issue returns 410 Gone on a single read, and a deleted repository simply stops appearing. Deletions reach you only through the issues and repository webhooks, so a scheduled sync needs a periodic full replace or a webhook listener.

Why do old workflow runs disappear from GitHub?

Since October 1, 2026 the GitHub Actions retention setting also governs checks, workflow runs, and statuses, and changing the setting does not restore data already removed. A sync that lands workflow runs in Postgres before the retention window closes is now the only way to keep CI history for lead-time and failure-rate metrics.

Can Postgres query GitHub without copying the data?

Yes, with the Steampipe GitHub plugin, which is Apache 2.0 and installs as a native Postgres foreign data wrapper with tables for issues, pull requests, reviews, commits, releases, stargazers, and workflow runs. Every query is an API call, so it suits ad hoc questions, and a scheduled copy is still the answer for dashboards.

Does Filament sync GitHub pull requests incrementally?

No. GitHub's pull request endpoint has no since filter, so Filament reads pull requests in full on every run and upserts them by id. Issues, issue comments, and review comments read incrementally on updated_at with a shared watermark and a 300-second overlap, and an interrupted full read resumes after its last completed page.

What GitHub token does Filament need?

A personal access token or a GitHub App installation token with read permission on the resources you select. Team listings need Organization Members read access, and stargazer and watcher identities need admin or collaborator access on the repository, so a narrow token returns empty lists for those resources rather than an error.