Machine Learning System for Text Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Unsupervised machine learning systems are not sufficiently accurate for deduplicating natural language text in databases due to high rates of false positives and false negatives, as they fail to correctly match entities with different names or identifiers.
Innovation Solution
A system that combines the outputs of multiple unsupervised machine learning models with a supervised metamodel to generate a final probability of text matching, using pre-processing normalization and distance metrics to improve accuracy in entity matching between databases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If an unsupervised machine learning system is used for text deduplication, then the system is suitable for natural language text processing, but the accuracy of matching results is insufficient with high false positive and false negative rates
Solution Approach 1:
The patent combines multiple machine learning systems (both unsupervised and supervised) into an integrated deduplication system. The unsupervised systems process natural language text while the supervised system refines the matching decisions, merging their capabilities to achieve both adaptability to natural language and high accuracy in matching results.
Solution Approach 2:
The patent creates a composite machine learning architecture that integrates different types of learning systems. By combining unsupervised learning components (for natural language processing) with supervised learning components (for accurate matching), the system achieves properties that neither component alone could provide, similar to how composite materials combine different substances to achieve superior properties.
2Measurement precision
If a supervised machine learning system is used for text deduplication, then the accuracy of matching results is improved, but the system is unsuitable for natural language text processing
Solution Approach 1:
The patent segments the deduplication system into distinct functional components: unsupervised machine learning systems that handle natural language text processing and a supervised machine learning system that handles accurate matching decisions. This segmentation allows each component to specialize in its strength while working together as a unified system.
Solution Approach 2:
The patent introduces an intermediary mechanism (the integrated system architecture) that bridges the gap between unsupervised and supervised learning systems. The unsupervised systems process natural language text and generate candidate matches, which are then refined by the supervised system, acting as an intermediary that translates between the two different learning paradigms.
3Measurement precision
If multiple machine learning systems are combined to improve matching accuracy, then false positives and false negatives are reduced, but the system complexity increases
Solution Approach 1:
The patent implements a dynamic system architecture where multiple machine learning systems operate in coordination. The system dynamically processes text through different learning models and integrates their outputs adaptively, allowing the complexity to be managed through dynamic control rather than static rigid structures.
Data Source
AI summary
A method including receiving first and second natural language texts. A distance metric is generated from the first and second natural language texts. A first machine learning system is executed, the first machine learning system taking, as a first input, the distance metric and generating, as a first output, a first probability that the first natural language text matches the second natural language text. A second machine learning system is executed, the second machine learning system taking as a second input the first natural language text and as a third input the second natural language text, and generating, as a second output, a second probability that the first natural language text matches the second natural language text. A third probability that the first natural language text matches the second natural language text is generated. Generating includes combining the first probability and the second probability.