Data Refinement Interface for Master Data Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current master data management processes are manual, time-consuming, and error-prone, particularly when dealing with large volumes of data from multiple sources, and existing ETL processing cannot automatically address data quality issues such as duplicates, misspellings, and format inconsistencies, requiring manual inspection and correction.
Innovation Solution
The implementation of user interfaces and heuristic algorithms that assist data stewards in identifying and refining data discrepancies and duplicates, with the ability to retain interactions for future use, and the use of statistical methods and visual displays to identify quality issues not addressed by ETL processing, applying refinement actions to create a 'best record' before data is released for consumption by business intelligence tools.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual review and inspection of records is performed to identify duplicates and data quality issues, then data accuracy can be improved, but time consumption and labor effort increase significantly
Solution Approach 1:
The patent segments the data consolidation process into distinct phases: automated ETL processing for initial data extraction and transformation, followed by selective manual review only for records that fail automated quality checks or are identified as potential duplicates. This segmentation allows most routine processing to be automated while reserving manual effort for edge cases, thereby maintaining data accuracy without requiring complete manual inspection of all records.
Solution Approach 2:
The patent introduces an intermediary automated quality assessment layer between ETL processing and final manual review. This intermediary layer uses automated algorithms to pre-screen records, identify potential issues, and prioritize those requiring manual attention. By acting as an intermediary filter, it reduces the volume of records needing manual inspection while ensuring that all critical quality issues are caught before final data consolidation.
2Productivity
If ETL processing is used for automated data extraction and transformation, then processing speed improves, but data quality issues such as duplicates, misspellings, and format inconsistencies cannot be addressed
Solution Approach 1:
The patent merges ETL processing with post-ETL quality validation and refinement steps into a unified data consolidation workflow. After ETL extracts and transforms data at high speed, the system automatically performs additional quality checks for duplicates, format consistency, and business rule compliance. Records failing these checks are flagged for targeted manual review, creating a hybrid approach that maintains processing speed while addressing quality issues that pure ETL cannot handle.
Solution Approach 2:
The patent implements feedback loops where results from automated quality validation feed back into the processing workflow. When quality issues are detected after ETL processing, the system automatically generates refinement actions and feeds them back to correct the data before final consolidation. This feedback mechanism allows the system to self-correct many quality issues without manual intervention, maintaining both speed and accuracy.
3Reliability
If complete manual inspection of all records is performed to ensure data quality, then data reliability improves, but productivity and processing efficiency decrease
Solution Approach 1:
The patent applies partial manual inspection rather than complete inspection of all records. Manual review is performed selectively on records that fail automated quality thresholds, are identified as high-risk for duplicates, or are flagged by the intermediary quality assessment layer. This partial action approach maintains data reliability for critical records while accepting that some low-risk records may be processed automatically without manual verification, thereby preserving processing efficiency.
Data Source
AI summary
A method for refining data discrepancies in data loads from multiple data stores includes receiving a data set containing data records from one or more data sources, comparing the received data set with a prior data set to identify delta records among corresponding records of the received and the prior data sets, recalculating a best record for data located in each of the corresponding delta records, and if a discrepancy exists in a data value for corresponding delta records, executing a refinement action rule. The refinement action rule includes at least one replace statement and search criteria. The method also includes capturing a user's interaction with a user interface presenting the data set, and storing the interaction in a data store as a refinement action rule. A system for implementing the method and a computer readable medium are also described.