Provenance, Lineage, and Auditability for AI-Ready Enterprise Context

Updated October 2026

AI-ready enterprise context is data an AI system can use and later defend. Lineage shows how datasets moved through pipelines. Provenance shows where each individual fact came from and why it holds its value. Auditability is the ability to rebuild a past answer from stored, versioned evidence. Most teams have some lineage, little provenance, and almost no audit trail for agent retrieval, which is where AI trust questions land.

Why is data access not enough for AI-ready context?

Access gives an AI system rows. It does not give the system any way to judge which rows to believe. A customer record in the CRM can disagree with the same customer in billing, and an agent has no signal for which value is authoritative unless someone stored that decision.

Definitions drift too. A metric that changed last quarter looks identical in a retrieval result unless it carries a version stamp. The result is three compounding risks. Automation becomes unreliable, wrong answers resist debugging, and decisions cannot be explained when a reviewer or regulator asks months later.

This piece is part of a series on enterprise context. The enterprise context strategy reference blueprint covers the full system. Here the focus is the evidence each layer must produce.

What is the difference between lineage, provenance, and auditability?

Lineage is about movement, provenance is about origin, and auditability is about reconstruction. Vendors often use the three words interchangeably. Treating them as one leads teams to buy a lineage tool and assume the trust problem is solved.

Data lineage

Lineage records how data flowed from sources through transformations to outputs. The OpenLineage object model is built around three entities, which are jobs, runs, and datasets. Run events are emitted at runtime when a job starts or finishes, while job and dataset events carry design time metadata such as source code location and schema.

Data provenance

Provenance is finer grained. The W3C PROV overview defines it as information about "entities, activities, and people involved in producing a piece of data". It is used to judge quality and trustworthiness. That covers a single attribute on a single entity, not only the pipeline that moved its table.

Auditability

Auditability is the operational ability to rebuild, inspect, and defend what happened at a past moment. It depends on lineage and provenance being stored durably and with versions. An auditable system can say what an agent retrieved at a given time, which definition it relied on, and whether the entity passed validation then.

For a broader comparison of catalogs, governance programs, and lineage tools, see data catalog vs data governance vs data lineage.

What does lineage capture well, and what does it miss?

Lineage answers pipeline questions very well. It shows which dbt model produced a table, which Spark job read it, and when the last run finished. OpenLineage is a graduate project of the LF AI and Data Foundation, with integrations for Airflow, Spark, dbt, Flink, and Trino. Marquez is "the reference implementation of the OpenLineage API" (OpenLineage docs).

Column lineage goes deeper. The OpenLineage column lineage facet maps each output column to its input columns. It labels each link as DIRECT, such as an aggregation, or INDIRECT, such as a join or filter. It can also flag whether an input was masked.

Platforms capture this natively as well. Databricks Unity Catalog records column level lineage automatically for queries run on Databricks, but loses it for renamed objects and path based tables. Its lineage system tables keep a rolling one year window. Snowflake keeps column and object lineage for one year on Enterprise Edition. It also accepts OpenLineage events from outside tools through a REST endpoint.

Where lineage stops

Lineage does not record which of three source records supplied a value on a merged entity. It does not keep the survivorship rule that chose one phone number over another. It does not know which ontology version defined a relationship when an agent used it. Those gaps matter most where entities are constructed rather than copied.

How do you build a layered evidence model for enterprise context?

Separate the evidence into four layers, each with its own standard and owner. Each layer answers a different question, and the audit layer links the other three together. Trying to stretch one tool across all four is the most common design mistake.

Provenance layer

PROV-O has been a W3C Recommendation since April 30, 2013. Its starting point classes are Entity, Activity, and Agent. Properties such as wasGeneratedBy, used, wasDerivedFrom, and wasAttributedTo link them. If your context layer runs on RDF, use PROV-O directly. If it runs on tables, mirror the same shape in columns so the meaning carries over.

For dataset level provenance, the Data and Trust Alliance Data Provenance Standards v1.0.0 define 22 metadata fields. They fall into three groups, which are source, provenance, and use. They work well as the header for any dataset that feeds an AI system. They do not replace fact level records.

This layer matters most for identity resolution and entity mastering, where merge rationale and survivorship decisions must stay inspectable.

Validation layer

The W3C SHACL Recommendation validates a data graph against a shapes graph. It returns a report with a conforms flag plus one result per violation. Each result names the focus node, the severity, and the constraint that failed. Store that report with the shapes graph version and a timestamp, and a reviewer can confirm an entity met the rules at a past moment.

The context QA gate for constraint validation covers how to run this control point in practice.

Audit and retrieval layer

Agent retrieval is the layer most teams skip. Each call should record the agent identity, the user it acted for, the tool or query, and the entity IDs and versions returned. The OpenTelemetry generative AI semantic conventions define spans, events, and metrics for model calls and MCP. That gives teams a shared format instead of a custom log.

What metadata should every governed entity carry?

Every governed entity and relationship needs a minimum evidence set. The exact schema varies, but the categories below cover most audit questions. Store references and versions, not copies of the data.

The meaning row depends on a governed ontology with owners and version IDs. Without that, the ontology version field has nothing stable to point at.

How do you trace one customer fact end to end?

A worked example makes the layers concrete. Imagine an agent that retrieves customer C-4821 to draft a contract renewal recommendation. The figures below are illustrative.

Before the merge

Three systems hold records for the same customer. The CRM record says the account is enterprise tier and names an account owner. The billing record says the contract is active with annual revenue of 2.4 million dollars. The support record shows a premium support plan. Each record lands with its source system, record ID, ingestion time, and run ID, which is the lineage layer doing its job.

During merge and modeling

Entity resolution decides the three records describe one customer and merges them. Survivorship rules take annual revenue from billing and tier from the CRM. The provenance layer stores which record supplied each attribute, which rule applied, and the match confidence. Ontology version 3.2 defines the account owner relationship. A rule derives renewal eligibility from an active contract and a premium plan.

At validation and serving time

Before publishing, SHACL checks confirm required properties exist and the owner relationship meets its cardinality. The report, shapes version 3.2.1, and timestamp are stored on the entity. Two weeks later the agent retrieves C-4821, and the retrieval log records the agent, user, query, entity version, and ontology version. A reviewer can now rebuild the full path from source records to the recommendation.

What do regulations and frameworks require?

The EU AI Act makes logging a legal duty for high risk systems. Article 12 requires "automatic recording of events (logs) over the lifetime of the system". Logs must support risk identification, post market monitoring, and deployer oversight. Article 19 and Article 26 require providers and deployers to keep those logs at least six months.

The dates moved in 2026. The Council gave final approval to the AI Omnibus on June 29, 2026. High risk rules now apply from December 2, 2027 for stand alone systems and August 2, 2028 for systems embedded in products. Article 10 separately requires data governance practices that cover data origin and preparation.

In the United States, the NIST AI Risk Management Framework is voluntary and is under revision. Its generative AI profile asks organizations to document data origin and content lineage, for example in action MP-2.1-001. ISO/IEC 42001 adds a certifiable management system around the same records. Legal counsel should map the specifics to your systems.

What are the most common failure modes?

Most teams that struggle with AI trust already have lineage. The failures below show up after the lineage tool is in place.

Missing fact level provenance

The team can trace a table to its pipeline but not an attribute to its source record. Every questionable answer becomes a manual hunt across several systems.

Unversioned definitions

Ontology terms and validation rules change, but entities built under old versions carry no stamp. A reviewer cannot tell whether a relationship reflects current logic or a retired definition.

Lost overrides and opaque merges

A steward fixes a value, but the prior value, reason, and approver are not kept. Entity resolution produces a golden record and throws away the merge rationale. In both cases the answer survives while the reasoning behind it disappears.

Retrieval logs that point nowhere

Agent traces capture prompts and responses but not the entity versions behind them. The log proves an answer happened without proving what it was based on. Our guide on data governance for AI agents covers agent identity and authorization in more depth.

Which design principles keep context auditable?

Retrofitting provenance after launch is expensive and always incomplete. Three principles are cheap to follow from the start.

Store evidence with the entity

Provenance and validation records belong next to the entities and relationships they describe. A separate audit store drifts out of sync. When a consumer retrieves an entity, its evidence should come with it or sit one lookup away.

Version everything that changes meaning

Ontology terms, metric definitions, validation shapes, and survivorship rules all need version IDs. Entities built under older versions should stay reconstructable. That is the difference between a context layer that supports audit and one that only shows the latest state.

Expose provenance at query time

Evidence that agents and APIs cannot see has little value. Retrieval paths should return source attribution, validation status, and version context on request. If the serving layer cannot say where a fact came from at query time, the chain breaks where it matters most.

This evidence is institutional knowledge, so it should live in open formats such as PROV-O, SHACL, and OpenLineage, on infrastructure the company controls. Galaxy is one partner that builds this kind of foundation on a team's existing stack, using internal tooling. It works in the open with its Apache 2.0 project Filament by Galaxy and plans to open source more.

Frequently asked questions

What is the difference between data lineage and data provenance?

Lineage records how datasets move through jobs and transformations, usually at the table or column level. Provenance records how a specific fact, entity, or relationship came to exist, including the source record, the process, and the person or system responsible. Lineage tells you which pipeline built a table. Provenance tells you why one customer attribute holds the value it does.

Why is lineage alone not enough for AI trust?

Lineage tools are built around datasets, jobs, and runs, so they answer pipeline questions well. AI trust questions usually arrive at a finer grain. A reviewer wants to know which source record supplied one attribute, which survivorship rule picked it, and which definition was active when an agent used it. Those answers need fact level provenance and versioned rules.

What makes an AI system auditable?

An auditable system can rebuild what happened at a past moment from stored evidence rather than memory. That means durable lineage, fact level provenance, versioned definitions, recorded validation results, and a log of what each agent retrieved and returned. If any one of those is missing, a reviewer can see the answer but cannot defend how it was produced.

What is OpenLineage?

OpenLineage is an open standard for collecting lineage events from data pipelines. Its model centers on jobs, runs, and datasets, extended through facets such as schema and column lineage. It is a graduate project of the LF AI and Data Foundation, and Marquez is its reference implementation. Airflow, Spark, dbt, Flink, and Trino all have integrations.

What is PROV-O and when should I use it?

PROV-O is the W3C ontology for provenance, a Recommendation since April 2013. It models entities, activities, and agents with properties such as wasDerivedFrom and wasAttributedTo. Use it when your context layer already runs on RDF or a knowledge graph, or when you need provenance that other systems can read without a custom schema.

Does the EU AI Act require audit logs?

Yes, for high risk systems. Article 12 requires automatic recording of events over the system lifetime. Providers and deployers must keep those logs for at least six months under Articles 19 and 26. After the 2026 AI Omnibus, these rules apply to stand alone high risk systems from December 2, 2027, and to product embedded ones from August 2, 2028.

What metadata should be stored on each entity?

Store the source system and record ID for every contributing record, plus ingestion and run identifiers. Add the ontology or schema version, the validation rule version and result, any policy tags, and the override history with approver and prior value. Merged entities also need the match confidence and survivorship rule. That set is enough to rebuild most audit questions.

How do I log what an AI agent retrieved?

Record each retrieval as an event with the agent identity, the user it acted for, the query or tool call, the entity IDs and versions returned, and the definitions in force. OpenTelemetry has generative AI semantic conventions for spans, events, and MCP calls, which gives teams a shared format. Link each event back to entity versions rather than copying data.

Can column level lineage replace fact level provenance?

No. Column lineage shows that a customer tier column was derived from a CRM column through a join or a transformation. It does not say which of three source records won for one customer, or why. Column lineage narrows the search. Fact level provenance answers the question for a single value and is what a reviewer usually needs.

Where should provenance metadata live?

Keep it next to the entities and relationships it describes, in the same governed context layer or knowledge graph. A separate audit store tends to drift out of sync. The serving layer should return provenance with each answer on request, so agents, analysts, and APIs can show where a fact came from at the moment they use it.