dbt freshness: Monitoring Data Staleness

dbt freshness: A Complete Guide

dbt freshness measures how up-to-date a source table is by validating its latest timestamp against warning and error thresholds you define in YAML.

What Is dbt freshness?

dbt freshness is a built-in validation that scans a source table’s loaded_at field and raises warnings or errors when data becomes older than expected.

Why Does dbt freshness Matter for Analytics SLAs?

Fresh data underpins reliable dashboards and machine-learning features. Automating staleness checks with dbt ensures teams catch upstream pipeline failures before stakeholders make decisions on outdated numbers.

How Does dbt freshness Work Internally?

During dbt run or dbt source freshness, dbt issues a single SQL query per source to fetch MAX(loaded_at_field). It then compares that timestamp to the current warehouse time and your YAML thresholds.

How Do I Configure dbt freshness Thresholds?

Add loaded_at_field and freshness blocks to your sources.yml. Set warn_after and error_after with time and count units so dbt knows when to alert.

YAML configuration example

sources:
- name: stripe
freshness:
warn_after: {count: 15, period: minute}
error_after: {count: 30, period: minute}
tables:
- name: payments
loaded_at_field: processed_at

How Can I Test dbt freshness Locally?

Run dbt source freshness --select stripe.payments. dbt prints a color-coded table, making it easy to validate thresholds before committing.

How Do I Surface dbt freshness in CI/CD Pipelines?

Include dbt source freshness in your CI job. Fail the pipeline on freshness errors and send Slack or PagerDuty alerts so incidents are triaged quickly.

Best Practices for Reliable dbt freshness Checks

Choose a loaded_at_field that updates only on successful ingest, align thresholds with business SLAs, and store freshness artifacts for observability dashboards.

Common Pitfalls and How to Avoid Them

Using a nullable timestamp, setting aggressive thresholds, or ignoring timezone conversions often leads to false alerts. Validate fields and timezones in staging first.

Galaxy Integration: Viewing dbt freshness Results in a Modern SQL Editor

After a dbt run, load the generated JSON artifacts into Galaxy. The IDE’s AI Copilot summarizes freshness failures and lets you drill into offending rows with one click, accelerating root-cause analysis.