Data Cleaning via Segmentation and Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data cleaning systems face challenges in scalability and quality when dealing with large volumes of data, as they often rely on minimal user intervention and do not effectively utilize multiple users or machine-based and human assets to address data errors and anomalies.
Innovation Solution
A computer-implemented method that partitions database tuples into fragments, detects errors, selects and assigns data cleaning assets based on error characteristics and capabilities, and reconciles candidate corrections using a confidence measure to provide a clean database instance, leveraging both human and machine-based cleaners in a crowdsourcing and parallel computing framework.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated cleaning algorithms are used with minimal user intervention, then productivity is improved, but data cleaning quality deteriorates
Solution Approach 1:
The patent segments the data cleaning process into multiple independent tasks that can be distributed to different cleaners (human or machine). Each cleaner works on a specific fragment of data, and their results are aggregated. This allows parallel processing (improving productivity) while maintaining quality through diverse cleaning approaches and user involvement.
Solution Approach 2:
The patent combines multiple cleaning results from different cleaners (human and machine) into a unified cleaning outcome. By merging diverse cleaning perspectives and approaches, the system achieves both high productivity (through parallel cleaning) and high quality (through combined expertise and error correction).
2Manufacturing precision
If a single user is involved in the cleaning process, then data cleaning quality is improved, but scalability deteriorates
Solution Approach 1:
The patent divides large datasets into manageable fragments and assigns them to multiple cleaners. This segmentation enables the system to scale to large data volumes by distributing the cleaning workload across many users or machines, while each individual cleaner can still work on smaller, more manageable portions with attention to quality.
Solution Approach 2:
The patent creates a universal cleaning framework that can handle both small and large datasets by dynamically allocating cleaners based on data volume and complexity. The system can use single users for complex, small datasets and scale to multiple users for large datasets, making the cleaning process adaptable to various scenarios.
3Manufacturing precision
If multiple cleaners are assigned to a fragment, then data cleaning quality is improved through redundancy, but device complexity increases
Solution Approach 1:
The patent implements feedback mechanisms where cleaning results from multiple cleaners are evaluated, compared, and aggregated. The system provides feedback to identify conflicts or inconsistencies in cleaning results and resolves them through predefined rules or additional user input. This feedback loop maintains quality while managing complexity through systematic conflict resolution.
Data Source
AI summary
A computer-implemented method comprising partitioning data representing an input instance of a database including multiple tuples into multiple fragments of tuples, detecting tuples which violate a data quality specification in respective ones of the fragments, selecting a data cleaning asset on the basis of characteristics of errors in detected tuples for a fragment and based on declared asset capabilities, assigning a selected data cleaning asset to the fragment, the selected data cleaning asset to provide a set of candidate corrections for the detected tuples in the fragment, providing data representing an output instance of the database in which detected tuples are replaced with selected candidate corrections.