Monitoring dbt Tests with Grafana: A Complete Guide

Monitoring dbt Tests with Grafana

dbt’s built-in tests generate artifacts that describe the health of every model in your warehouse. By piping those artifacts into Grafana you can create always-on observability: dashboards, alerts, and SLAs that keep bad data from ever reaching production consumers.

Why Combine dbt and Grafana?

dbt focuses on transforming data and validating assumptions at build time; Grafana excels at real-time visualization and alerting. Together they provide a full feedback loop for data quality:

• Immediate visibility into failing tests across environments.
• Historical trend analysis to spot flaky data sources and regressions.
• Alerting & on-call routing so the right engineer is paged before dashboards break.
• Self-service insights for analysts and product managers who lack CLI access to dbt Cloud/Core.

High-Level Architecture

The most common pattern looks like this:

1. Run dbt (Cloud job or CI). Enable --write-json so run_results.json and manifest.json are produced.
2. Parse artifacts with a lightweight script (Python, Bash, or dbt built-in state operations) that extracts test status, model name, and timestamps.
3. Expose metrics to Prometheus via a Pushgateway, StatsD, or a custom exporter.
4. Visualize & alert in Grafana using PromQL or the new Metrics Data Source.

If you already run your transformation jobs in Kubernetes or Airflow you can attach a Prometheus side-car and skip step 3.

Step-by-Step Implementation

1. Capture dbt Artifacts

Add the following flags to every dbt invocation:

dbt run --write-json
# or for tests
dbt test --write-json

The files will land in ./target/. To keep history, copy them to durable storage (S3, GCS) with a job ID in the path.

2. Parse and Emit Metrics

The minimal information you need for Grafana:

• test_name
• status (pass/fail/error/skip)
• execution_time (seconds)
• invocation_id or run_started_at

Python example that converts run_results.json into Prometheus gauge metrics:

import json, time, prometheus_client
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

registry = CollectorRegistry()
status_gauge = Gauge(
"dbt_test_status",
"dbt test status (1=pass,0=fail)",
["test_name"],
registry=registry,
)
exec_gauge = Gauge(
"dbt_test_execution_seconds",
"Runtime of each dbt test run",
["test_name"],
registry=registry,
)

with open("target/run_results.json") as f:
results = json.load(f)["results"]

for r in results:
metric_value = 1 if r["status"] == "pass" else 0
status_gauge.labels(test_name=r["unique_id"]).set(metric_value)
exec_gauge.labels(test_name=r["unique_id"]).set(r["execution_time"])

push_to_gateway("https://prometheus-push:9091", job="dbt_tests", registry=registry)

3. Build Grafana Dashboards

Once Prometheus scrapes or receives the pushed metrics:

• Create a new panel with query sum by (test_name) (dbt_test_status == 0) to show currently failing tests.
• Add a heatmap: rate(dbt_test_status[1d]) to track flakiness.
• Configure alert rules: “If failing tests > 0 for 5 minutes, send PagerDuty.”
• Use templating variables ($environment, $project) to slice by workspace.

4. Close the Loop

When an alert fires:

1. On-call engineer clicks the Grafana link.
2. Panel shows failing test ID. The ID maps 1-to-1 with the dbt test file.
3. The engineer runs dbt test --select <failing_test> locally or in Galaxy’s SQL editor to debug.
4. Fix issue, merge PR, next run resets the metric to 1.

Best Practices

Use Incremental and Source Freshness Tests

Emitting only generic schema tests (not_null, unique) misses upstream freshness issues. Include source_freshness run in the same job and send that to Grafana as well.

Normalize Metric Names

Prometheus recommends lowercase, snake_case, and units. E.g. dbt_test_execution_seconds, not DbtTestExecTimeMs.

Display Context

Add labels for env, team, and model_name so dashboards remain decipherable as projects grow.

Version Control the Dashboards

Export dashboards as JSON and commit to Git. Use Grafana’s Terraform provider or Grafana Loki for GitOps-style deployments.

Common Mistakes and How to Fix Them

Misreading status Values

Problem: Treating error the same as fail inflates incident count.
Fix: Map error to -1, fail to 0, pass to 1, and alert only on 0.

Overloading Pushgateway

Problem: Pushing thousands of metrics every minute leads to high cardinality and OOM errors.
Fix: Push once per job, delete old time-series with metric_relabel_configs, or switch to a remote_write-compatible backend.

Ignoring Historical Context

Problem: Only current status is displayed; patterns are missed.
Fix: Store artifacts in S3 and backfill metrics with a nightly batch job. Visualize 30-day MTTR (mean time to resolution).

Galaxy and dbt/Grafana Workflow

Galaxy’s desktop SQL editor is the fastest way to reproduce a failing dbt test:

1. Copy the failing model’s SELECT from the dbt model file.
2. Paste into Galaxy; its AI copilot highlights potential null columns or duplicate keys.
3. Iterate until the query passes, commit changes to Git, and rerun the pipeline.

Because Galaxy keeps a history of every query, you can track what changed between the failing and passing versions.

Conclusion

By exporting dbt test artifacts to Prometheus and visualizing them in Grafana, teams gain real-time insight into data health, shorten incident resolution time, and build trust in their analytics layer. The approach is lightweight, open-source friendly, and scales from a single cron job to thousands of daily runs. With tools like Galaxy accelerating SQL diagnostics, the feedback loop becomes even tighter, ensuring that data engineers catch quality issues long before they reach stakeholders.