ML Data Matching via Steward Feedback Loops
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Companies face significant challenges in managing customer data due to frequent changes, inconsistencies, and errors, especially when data is sourced from multiple departments and stored across different cloud providers, leading to complex data management issues and potential errors in matching and clustering records.
Innovation Solution
A machine learning framework that leverages feedback from data stewards to improve data matching by revising factors, offsets, and weights for fields, enabling more accurate record matching and clustering by determining factored and weighted scores based on steward feedback, and creating training instances to learn from false matches and missed matches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a database system automatically matches and clusters customer records, then processing speed and productivity are improved, but matching accuracy and reliability deteriorate due to errors in identifying similar records
Solution Approach 1:
The system implements feedback loops where data stewards review and correct matching decisions. The feedback mechanism allows stewards to override automatic matches, report false positives and negatives, and provide correction data that is used to retrain the machine learning model, continuously improving matching accuracy while maintaining automated processing
Solution Approach 2:
A hybrid matching system acts as an intermediary between fully automated matching and manual review. The system automatically processes records but introduces an intermediate review stage for borderline cases, combining the speed of automation with the accuracy of human judgment
2Reliability
If data stewards manually review and correct matching decisions, then matching accuracy is improved, but processing time and operational complexity increase
Solution Approach 1:
Instead of requiring manual review of all records, the system applies partial manual action only to borderline cases that fall within a confidence threshold range. High-confidence automatic matches are processed without human intervention, while only uncertain cases require steward review, minimizing time loss
Solution Approach 2:
The machine learning model performs preliminary matching and filtering before human review, pre-processing the data to identify only those cases requiring manual attention. This preliminary action reduces the volume of work for data stewards and accelerates the overall process
3Quantity of substance
If the system processes data from multiple sources and departments, then data completeness and coverage are improved, but data consistency and management complexity worsen
Solution Approach 1:
The machine learning model serves multiple functions simultaneously: it matches records across different data sources, clusters duplicate records, identifies data quality issues, and learns from various types of feedback. This universal approach handles diverse data sources through a single unified system rather than separate processes for each source
Solution Approach 2:
The system dynamically adjusts matching parameters and thresholds based on the specific characteristics of different data sources and feedback received. Matching criteria are not fixed but adapt to the varying quality, format, and characteristics of data from different departments and cloud providers
4Reliability
If the machine learning model continuously learns from feedback, then matching accuracy is improved, but computational resource requirements and system complexity increase
Solution Approach 1:
The system performs continuous learning through periodic retraining cycles rather than constant real-time retraining. Feedback is accumulated over time and the model is retrained at scheduled intervals or when sufficient feedback data is gathered, reducing computational burden while maintaining continuous improvement
Data Source
AI summary
A system determines factored score by multiplying factor and match score for values of field in two records, offset score by adding offset to factored score, and weighted score by applying weight to offset score. The system determines status for two records based on combining weighted score with other weighted score corresponding to other field of two records. The system revises factor, offset, and weight based on feedback associated with two records. The system determines revised factored score by multiplying revised factor and match score for other values of field in two other records, revised offset score by adding revised offset to revised factored score, and revised weighted score by applying revised weight to revised offset score. The system determines learned status for two other records based on combining revised weighted score with additional weighted score corresponding to other field for two other records.