Whenever a customer interacts with a business, a unique record is logged in the business’s database. Sometimes, this results in two different data points such as “T. Cruise” and “Tom Cruise” referring to the same Tom Cruise. If you want to get a more accurate, comprehensive view of customer behavior to improve your customer service and marketing, you may need to resolve these two points into one distinct entity. This is one example of what is called entity resolution, the process of merging various records into a single entity.
What is Entity Resolution?
To start, let’s define what an entity is—something relevant to a business that has a certain identity, such as customers, products, or even orders. Entity resolution creates clear and consistent views of business entities by stitching together data from various sources, both internal and external. Internal data is information created by your organization; if we use the example of an individual customer, this could be transactions or communications you’ve had with that customer. External data is sourced from third party tools, such as corporate registries; onboarding this data may require importing and converting it into a format or structure compatible with your system. Combining these different data sources for a customer creates one clear picture so you can accurately analyze their activity.
Beyond customers, the final entity that is produced can also be larger units of your consumer base, like entire households. As an example, consider an internet provider that logged a conversation from one family member with their online customer service chat feature, then received a call from another family member the next day. Merging those two family members into one household provides a more holistic understanding of their needs, allowing the agent taking the call to provide more helpful information. For business-to-business companies, entities can also be other companies or teams. In any case, the goal is a unified record that contains all the relevant information about that entity, eliminating duplicates and inconsistencies.
Why Entity Resolution Matters
Improving the quality of your data has implications in your business strategies. Although each individual inconsistency may seem insignificant, resolving them makes your data more accurate and thus more valuable for analysis and decision-making. This can lead to better sales strategies and targeted marketing since you now have a clearer view of your customers; in turn, this optimization can help reduce the costs of your marketing and maximize your revenue. As aforementioned, entity resolution can also help provide a more comprehensive view of customers, leading to better service and customer satisfaction. And if you ever want to implement machine learning and AI in your business strategies, clean data is essential for making your predictive models accurate.
More practically, if inconsistent data builds up over time, it will likely cost exponentially more the longer you wait to correct it. Entity resolution helps correct data records as they are ingested, saving this money in the long run. Additionally, there are official regulations in place for companies that require accurate handling of personal data, such as the Know Your Client (KYC) standard for the investment industry or the General Data Protection Regulation (GDPR) in the European Union. If you want to keep your business, it is in your best interest to prioritize accurate identification and linking of your customer records to comply with these regulations.
How Does Entity Resolution Work?
There are four general steps to entity resolution: ingestion, deduplication, record linkage, and canonicalization. These steps can get a bit technical, so let’s use a metaphor of a toy box to help make it more intuitive.
Ingestion
Ingestion is concerned with gathering all the data necessary to begin the process of identifying and resolving duplicates and inconsistencies; think of it like cleaning up and gathering all the toys that belong to your toy box. The process of retrieving data from their respective sources can involve querying databases, using APIs to access external data like censuses, or extracting from Excel files. You then must clean and standardize this data to a format suitable for the entity resolution system, which is a single location that stores all this data so it can be more easily handled, such as a data warehouse, a database, or a specialized data processing application. It is crucial that you have the technology in place to handle these large volumes of data and integrate it from disparate sources.
Deduplication
Suppose you have several rubber duckies, and you really want only one rubber ducky in your toy box because why would you need more? Deduplication refers to the step of identifying copies and merging them into a single, unified record, while simultaneously resolving conflicts in data for a single entity (e.g. two different phone numbers for an individual customer). For this step, it is important to strike a balance between identifying all duplicate records and not incorrectly merging distinct records.
Record Linkage
Now your nice friend just brought over their toy box and offers to combine it with yours—but you still only want one rubber ducky. Whereas deduplication is used within a single data set, record linkage is used to recognize records that refer to the same entity across data sets, connecting information that is stored in various places. For example, you may want to recognize that two different interactions on two different days belong to the same customer. Record linkage is more complex than deduplication, as you must account for a greater variation in data format.
Consolidation
Lastly, consolidation is the process by which you collect and store all the deduplicated and linked records relevant to a single entity. Again, this data must be standardized in anticipation of further analysis. Ah, now you can relax knowing that all your toys have been gathered in one place and you don’t have to worry about spare rubber duckies.
Deterministic vs. Probabilistic Entity Resolution
For the deduplication step in entity resolution, there are two main methods: deterministic and probabilistic. Deterministic entity resolution, also known as rules-based matching, uses predefined rules and criteria to determine matches between records. These rules are established based on fields within the data records; for example, two records might be deemed a match if they have the same name, address, and date of birth. Because these rules must be strictly applied and often require exact matches on specific fields, this method is best for data that is consistently formatted and higher-quality.
This rigidity makes deterministic entity resolution easier to implement and thus faster in execution; it can be implemented using specialized software or even a custom script written in Python or SQL. However, the simplicity of the rules can clearly also become a limitation if data becomes more complex or variable. Additionally, it is possible that occasional outliers or ambiguities like typos fly under the radar, which makes this method less reliable.
This is where probabilistic entity resolution comes in. Instead of rules, it relies on machine learning, AI, and predictive algorithms to find the probability that any two records belong to the same entity, a technique called fuzzy matching. If this probability falls within an accepted level of statistical confidence, the records will be merged. Instead of names and addresses, probabilistic entity resolution may consider data like IP addresses and device operating systems. This approach allows companies to make connections between data even when the quality is low, records are stored in different formats and locations, or the data has been manipulated. However, the messier nature of the data may still result in a lower degree of accuracy.
Because both methods have their advantages and drawbacks, many entity resolution strategies synthesize these two methods into a two-step process: first filter out obvious inconsistencies with deterministic entity resolution, then handle more complex cases with probabilistic entity resolution.
Implementing Entity Resolution
As technology evolves, it’s inevitable that companies are going to rely more and more on larger volumes of data to inform and optimize their business strategies. It’s inevitable that entity resolution can get computationally intensive, making maintaining a standard of accuracy difficult—especially if the data is inconsistent or sourced from a variety of places.
Incorporating machine learning algorithms into your approach to entity resolution can help automate the process, allowing you to make faster and more accurate decisions. It can also use the data to help identify risks you might have missed or hidden opportunities to grow. With Cotera’s AI models, you can expect to expedite every step of entity resolution and provide key analysis on your company’s data, revealing key insights you can use to improve and optimize your marketing efforts.
Conclusion
Entity resolution is a crucial first step for companies to truly understand the habits and needs of customers in order to improve their products. With the help of third-party algorithms like the ones Cotera can provide, you can obtain a truly holistic view of the condition of your business, as well as gain insight on strategies you should implement in the future. Who would’ve thought cleaning toy boxes could be so useful?