What is Entity Resolution?

Updated October 2026

Entity resolution is the process of deciding which data records refer to the same real-world thing. It compares records that lack a shared ID, scores how likely each pair is to match, and groups the matches under one identifier. Teams use it to merge duplicate customers, link patients across hospitals, and connect accounts in fraud investigations.

What is entity resolution?

Entity resolution turns many partial records into one known entity. A customer might appear as "Robert Johnson" in billing, "Bob Johnson" in support, and "R. Johnson" in a loyalty app. No field is shared exactly, yet a person would say they are the same. Entity resolution makes that call systematically and at scale.

The idea is old. Halbert Dunn introduced the term record linkage in a 1946 public health article, according to the Wikipedia overview of record linkage. The same overview lists the many other names in use, including data matching, deduplication, duplicate detection, and fuzzy matching. Healthcare teams say patient matching, and marketers say identity resolution.

Senzing's definition adds a useful detail. Resolution decides when records refer to the same entity and also when they do not. A false merge can be worse than a missed match, because it mixes two people's histories.

Why does entity resolution matter?

Every count, model, and agent answer depends on knowing what is one thing. If one customer is three records, revenue per customer is wrong and churn models see three thin profiles. If two different patients share one record, a clinician may see the wrong allergy list.

Healthcare shows the stakes clearly. A Pew Charitable Trusts report found match rates can be as low as 80 percent inside one organization. Between organizations, rates fell as low as 50 percent, even when both used the same EHR vendor.

AI raises the bar further. A language model or agent that queries your data cannot tell that two rows are one supplier unless something has already resolved them. That is why resolved entities sit at the base of ontologies that power AI analytics and of most knowledge graphs.

How does entity resolution work, step by step?

Most pipelines follow the same five stages, whatever the tool. The stages are standardize, block, compare, score, and cluster. Each one trades accuracy against cost, so tuning happens at every stage rather than only in the model.

1. Standardize. Clean and normalize fields, such as lowercasing names, parsing addresses, and formatting phone numbers.

2. Block. Pick candidate pairs that share something, so you never compare every record with every other record.

3. Compare. Compute similarity per field, using exact equality, edit distance, Jaro-Winkler, phonetic codes, or embeddings.

4. Score. Turn field comparisons into a match probability with rules, a statistical model, or a classifier.

5. Cluster. Group pairs above a threshold into entities and assign each a stable ID.

What is deterministic matching?

Deterministic matching links records when chosen fields agree exactly or after normalization. A rule might say that two records match if the tax ID agrees, or if email and date of birth both agree. The record linkage overview calls this rules-based linkage and notes that it counts how many identifiers match.

Rules are fast, cheap, and easy to explain to an auditor. They fail when data is dirty. A single typo in an email or a missing date of birth means no match, so recall drops as data quality drops. Rules also need ordering, because a strict rule should fire before a loose one.

AWS Entity Resolution shows how vendors package this. Its rule-based matching lets you add, remove, and reorder rules, and each match records which rule produced it. Rule 1 is treated as more precise than Rule 2.

How does probabilistic matching and the Fellegi-Sunter model work?

Probabilistic matching weighs evidence from each field instead of demanding exact agreement. Ivan Fellegi and Alan Sunter formalized it in 1969 in "A Theory for Record Linkage." The record linkage overview notes their decision rule is optimal when comparison fields are conditionally independent.

The model uses two numbers per field. The Splink guide to Fellegi-Sunter defines the m probability as the chance a field agrees given a true match. The u probability is the chance it agrees given a non-match, which captures coincidence.

The log of m divided by u becomes a match weight for that field. Agreement on a rare surname carries a large weight, while agreement on a common one carries little. Weights from all fields add together with a prior, and the total converts into a match probability between 0 and 1.

The method is equivalent to Naive Bayes, per the same overview. Its weakness is the independence assumption. City and postcode are strongly correlated, so counting both can overstate the evidence. Modern tools add term frequency adjustments and other fixes on top of the base model.

How do machine learning and LLMs change entity resolution?

ML models learn which field patterns predict a match, and LLMs can judge pairs by reading them. Supervised classifiers such as random forests and SVMs often beat classic probabilistic scoring when labeled pairs are available, according to the record linkage overview. The cost is labeling, which tools reduce with active learning.

Large language models are the newest option. Researchers at the University of Mannheim tested them in Entity Matching using Large Language Models. GPT-4 reached over 89 percent F1 on five of six datasets with no task-specific training. It beat fine-tuned RoBERTa and Ditto on three of four e-commerce datasets.

The same study found fine-tuned PLM matchers lost 22 to 61 percent F1 on unseen entities, while LLMs held up better. Fine-tuned Llama 3.1 models matched or beat zero-shot GPT-4 on four of six datasets. Smaller models were far more sensitive to prompt wording than GPT-4.

In practice LLMs sit late in the pipeline. Running a model call on billions of pairs is not affordable. Teams block first, score with a cheap model, and send only the uncertain middle band to an LLM for a decision or an explanation.

What is blocking and why is it needed?

Blocking keeps the number of comparisons manageable by only comparing plausible pairs. The Splink blocking guide notes that pairs grow as n times n minus 1 over 2. One million records means roughly 500 billion comparisons, which is not practical to score.

A blocking rule might compare only records that share a first name and surname, or a postcode. Splink recommends several strict rules over one loose rule. If a surname has a typo, a postcode rule can still catch the match.

Tools differ in how they build blocks. The dedupe library uses predicates such as the first three characters of a field, then picks a small rule set with a greedy set cover algorithm. Zingg learns a blocking model that typically cuts comparisons to 0.05 to 1 percent of all pairs.

Which entity resolution tools are current in 2026?

The field splits into open-source libraries, cloud services, embedded engines, and enterprise platforms. The table lists the main options with a documented strength and limit for each. For a deeper look at running these inside an enterprise program, see the companion guide to entity resolution for enterprise data.

Splink

Splink is a Python package from the UK Ministry of Justice that implements Fellegi-Sunter with customizations. It runs on DuckDB by default and on Spark, PostgreSQL, and SQLite, per PyPI, where version 5.0.0 shipped on September 28, 2026. The project says it can link a million records on a laptop in about a minute. You still write the comparison and blocking logic yourself.

Zingg

Zingg is an ML-based matcher that learns both blocking and similarity models from a small labeled set built through active learning. It runs on any Spark service and reads from Snowflake, S3, and major databases. The open-source edition is AGPL 3.0, and Zingg also sells an enterprise version on Snowflake.

dedupe

dedupe is a mature MIT-licensed Python library built on learnable similarity functions and active learning. It suits small and mid-sized jobs well. The latest release on PyPI is 3.0.3 from August 15, 2024, so check activity before betting a new system on it.

Senzing

Senzing is an embedded SDK that uses principle-based matching instead of trained models, so new data sources work without tuning. It resolves records in real time and tracks relationships between entities. Pricing starts at 58,560 dollars a year for 10 million data source records, and only 500 records are free.

AWS Entity Resolution

AWS Entity Resolution offers rule-based matching, a prebuilt ML model tuned for consumer records, and matching through licensed data providers. The ML model returns confidence scores from 0 to 1. Pricing is 0.25 dollars per 1,000 records for rules and ML, with every processed record billed. Incremental processing works only for rule-based matching.

Tamr and Quantexa

Tamr calls itself an AI-native MDM platform, with entity resolution, an enterprise knowledge graph, and MCP connectivity for LLMs. It fits teams that want matching, stewardship, and golden records in one product, but pricing is not public. Quantexa rebuilds entities dynamically from raw data and pairs them with graph analytics for fraud and risk. Its accuracy figures are vendor reported.

Google's Entity Reconciliation API is also worth a look for BigQuery teams. It is still in Preview and supports up to 10 BigQuery tables.

Where is entity resolution used?

Any domain where the same thing shows up in many systems needs it. The use cases below are the most common, and each leans on a different technique.

Public sector and research linkage

Government data rarely shares a person ID across departments. The UK Ministry of Justice built Splink to link justice systems without one, and the UK government notes it has handled about 15 million records in under an hour. The ONS peer reviewed the work.

Fraud, AML, and investigations

Fraud rings reuse phones, addresses, and devices across many identities. Resolving those attributes reveals that separate accounts belong to one network. Quantexa announced a 10-year project with HMRC in May 2026 to find fraud and tax return errors.

Customer 360 and product data

Marketing and support teams resolve customers across CRM, ecommerce, and support tools. Product teams resolve supplier SKUs, where titles are messy free text. That free-text case is where the Mannheim LLM study showed the largest gains over older models. For the customer case end to end, see building a single customer view.

Knowledge graphs and AI context

A knowledge graph with duplicate nodes gives wrong answers with high confidence. Resolution should happen before data lands in the graph, which is why knowledge graph platforms and MDM tools both ship matching features.

How is entity resolution different from MDM and identity resolution?

Entity resolution is the matching step, and MDM is the program around it. Master data management adds survivorship rules, stewardship, and syncing golden records back to sources. Identity resolution usually means the customer and marketing flavor of the same task.

Resolved entities are also institutional knowledge. Match rules, thresholds, and stable IDs encode years of learning about your customers and suppliers. They should live in open formats on infrastructure you control, so you can audit them and reuse them across BI, search, and AI agents.

Galaxy is an applied lab that builds this layer with teams on the stack they already run. It uses internal tooling to stand up entity resolution, ontologies, and semantic models quickly, builds in the open with Filament by Galaxy, and plans to open source more of that tooling.

Frequently asked questions

What is entity resolution in simple terms?

Entity resolution is the work of deciding which records describe the same real person, company, product, or place. It compares records that have no shared ID, scores how alike they are, and groups the ones that match. The result is one stable identifier per real thing, which every report, model, and agent can then use.

Is entity resolution the same as record linkage?

Mostly yes. Record linkage is the older academic name, coined by Halbert Dunn in 1946. Entity resolution, data matching, deduplication, and merge purge describe the same core task. Some teams use entity resolution for the full pipeline and record linkage for the pairwise matching step inside it, but the methods are shared.

What is the Fellegi-Sunter model?

It is the classic probabilistic model for record linkage, published by Ivan Fellegi and Alan Sunter in 1969. Each field gets an m probability, the chance it agrees on true matches, and a u probability, the chance it agrees by coincidence. The log ratios add up to a match weight that becomes a match probability.

What is blocking in entity resolution?

Blocking limits which record pairs get compared. Comparing every pair grows with the square of the data, so one million records means about 500 billion pairs. Blocking rules only compare records that share something, such as a postcode or a surname. Using several strict rules lets one rule catch matches another misses.

Should I use deterministic or probabilistic matching?

Use deterministic rules when you have a trusted shared identifier, such as a tax ID or verified email, and you need decisions you can explain line by line. Use probabilistic or ML scoring when identifiers are missing or messy. Most production pipelines run exact rules first and send the leftover records to a scoring model.

Can large language models do entity resolution?

They work well as matchers on hard pairs. A University of Mannheim study found GPT-4 beat fine-tuned models on three of four e-commerce datasets with no task-specific training. LLMs are slow and costly per comparison, so teams use them after blocking and cheaper scoring, for the uncertain middle band of pairs.

What are the best open-source entity resolution tools?

Splink is the most active option in 2026, with version 5.0.0 released in September 2026 under the MIT license. Zingg runs on Spark under AGPL 3.0 and learns its own blocking. The dedupe Python library uses active learning under MIT, though its last release was 3.0.3 in August 2024.

How much does AWS Entity Resolution cost?

AWS lists 0.25 dollars per 1,000 records processed for rule-based and ML matching, and 0.10 dollars per 1,000 records for data service provider matching, which also needs a provider license. Every processed record is billed, including records that find no match, and there is no free tier.

How is entity resolution different from master data management?

Entity resolution answers which records are the same thing. Master data management goes further, deciding which values survive into a golden record, who governs changes, and how the result syncs back to source systems. Entity resolution is usually the matching engine inside an MDM program, but you can run it without one.

Who should own the resolved entities?

The company should own them. Resolved IDs, match rules, and model weights encode what the business knows about its customers and suppliers. Keeping them in open formats on infrastructure you control means you can audit decisions, switch tools, and reuse the same entities across analytics, search, and AI agents.