Entity Name Resolution Using Hybrid Search and LLM Candidate Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional entity resolution techniques struggle with incomplete, inconsistent, or incorrect data, particularly in contexts where entity names exhibit variations such as misspellings, abbreviations, and differing naming conventions, leading to inefficiencies in systems like identity management and fraud detection.
Innovation Solution
A system utilizing large language models (LLMs) and advanced algorithms for entity resolution, combining sparse and dense vector embeddings, hybrid search techniques, and virtual entity generation to enhance accuracy and efficiency in identifying and matching entity records, even with noisy or incomplete data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional entity resolution techniques are used, then the system is simpler to implement, but accuracy deteriorates when dealing with noisy or incomplete data
Solution Approach 1:
The system segments the entity resolution process into distinct phases: indexing phase (where data is processed and stored) and querying phase (where resolution occurs). This segmentation allows each phase to be optimized independently, improving overall accuracy while managing complexity through modular architecture.
Solution Approach 2:
The patent introduces a virtual entity generator as an intermediary component that creates synthetic entity records when real entities cannot be found. This mediator handles the complexity of dealing with noisy or incomplete data by generating placeholder entities that can be processed and resolved, thereby improving accuracy without requiring the entire system to handle every edge case directly.
2Measurement precision
If hybrid search with multiple embedding models is used, then search accuracy improves, but processing time increases
Solution Approach 1:
The system performs preliminary action by pre-computing and storing both sparse and dense vector embeddings during the indexing phase. This allows that during the querying phase, the system can quickly retrieve pre-processed data without having to compute embeddings in real-time, thus reducing processing time while maintaining the accuracy benefits of hybrid search.
Solution Approach 2:
The patent applies partial action by selectively using different search strategies based on the query characteristics. The system can choose to use only sparse vector search for simple queries, only dense vector search for semantic queries, or both for complex queries, thereby optimizing processing time while maintaining high accuracy through partial application of the most effective search methods.
3Reliability
If virtual entity generation is employed, then handling noisy data improves, but computational resources increase
Solution Approach 1:
The virtual entity generator operates as a self-service component that automatically creates synthetic entities when real entities cannot be found. This self-service mechanism handles noisy data by generating appropriate placeholder entities without requiring manual intervention or extensive computational resources, as the system uses efficient template-based generation rather than complex real-time analysis.
Data Source
AI summary
Systems and methods for resolving entity names combine hybrid search techniques with large language models to improve accuracy in matching business entities. The system includes an index builder that generates and stores both sparse and dense vector embeddings representing entity information. A query component performs hybrid searches using both keyword-based and semantic methods to retrieve entity candidates from the stored embeddings. When high-confidence matches are not found, a virtual entity generator leverages large language models to generate additional entity candidates based on the input data. The system employs a re-ranker to order retrieved candidates based on their relevance to the input query. This approach enables accurate entity resolution even with noisy or incomplete input data by combining traditional search methods with advanced language model capabilities.