Record Clustering Using Imperfect Rules and ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication methods are inefficient in utilizing heuristic human knowledge and require extensive management of rule priorities and conflict resolution, especially when dealing with large datasets.
Innovation Solution
Incorporating rules to represent heuristic human knowledge alongside traditional labeled examples at multiple stages of the machine learning workflow, allowing for the creation of a machine learning model using fewer rules than point examples, and enabling imperfect and contradictory rules without compromising accuracy, while simplifying the management of large collections of heuristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If traditional rule-based systems are used to represent heuristic knowledge, then knowledge can be explicitly encoded, but detailed management of rule priority and conflict resolution is required
Solution Approach 1:
The patent introduces machine learning models as an intermediary layer between heuristic rules and data deduplication decisions. The rules serve as training data rather than direct decision-making agents, allowing the system to benefit from structured knowledge without requiring complex rule management. The ML model learns from rule-based training examples and automatically handles conflicts through learned patterns.
Solution Approach 2:
The patent replaces the mechanical rule-based decision system with a machine learning-based system. Instead of explicitly managing rule priorities and conflicts through mechanical means (if-then logic), the system uses learned patterns from training data that includes rule-based examples, allowing for more flexible and adaptive conflict resolution.
2Adaptability or versatility
If more rules are added to represent comprehensive heuristic knowledge, then coverage of edge cases improves, but conflict resolution becomes more difficult
Solution Approach 1:
The patent applies preliminary action by pre-processing heuristic knowledge into structured training examples before the ML model is deployed. Rules are converted into labeled training data that captures edge cases and conflicts, allowing the model to learn resolution strategies in advance rather than requiring complex runtime conflict management.
Solution Approach 2:
The patent changes the parameter representation of heuristic knowledge from explicit rules to training data examples. This transformation allows the system to handle conflicts by learning from diverse examples rather than managing rule conflicts directly, effectively changing how knowledge is stored and utilized.
3Measurement precision
If machine learning models are trained with extensive labeled examples, then accuracy improves, but the process becomes less efficient compared to rule-based approaches
Solution Approach 1:
The patent merges rule-based approaches with machine learning by using rules to generate training data. This combination allows the system to leverage the efficiency of rule-based knowledge encoding while achieving the accuracy benefits of machine learning, as the rules provide structured examples that guide the ML model's learning process.
Solution Approach 2:
The patent applies preliminary action by pre-generating training data from rules before model training. This allows the system to efficiently create high-quality training examples using rule-based logic, reducing the manual effort required for data labeling while maintaining high model accuracy.
Data Source
AI summary
Record clustering is performed for a collection of records using training rules, training-rule labels, training data created from a sample of pairs of records, a pair-wise classifier, and a clustering algorithm. Record clustering is also performed for a collection of records using prediction rules, prediction-rule labels, a pair-wise classifier, and a clustering algorithm.