Priority-Based Data Cleaning for Lower-Cost Completeness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data cleaning tools are inefficient as they fail to prioritize portions of data that should be cleaned to achieve a reasonable data completeness score while reducing computational costs.
Innovation Solution
A method for prioritized data cleaning that involves determining prioritized features and scores, generating models for data resolution algorithms, and applying these algorithms in an ordered manner to resolve data issues based on priority, allowing users to select and compare different cleaning pathways.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data cleaning is performed on entire dataset without prioritization, then data completeness score is improved, but computational costs increase
Solution Approach 1:
The patent segments the dataset into different priority levels (high, medium, low priority features) and processes them in order of importance. This allows the system to focus computational resources on cleaning high-priority features first, achieving acceptable data completeness without processing the entire dataset, thereby reducing overall computational costs.
Solution Approach 2:
The patent applies partial action by cleaning only the necessary portion of the dataset (high and medium priority features) rather than the entire dataset. The system determines when sufficient data completeness is achieved and stops processing, avoiding excessive computation on low-priority features that may not significantly impact overall data quality.
2Reliability
If multiple data resolution algorithms are applied to all data issues, then data quality is improved, but processing time increases
Solution Approach 1:
The patent segments both the data issues and resolution algorithms into priority-based groups. It generates separate models for each algorithm indicating computing costs, then applies algorithms in an ordered manner based on feature priority. This segmentation allows the system to select appropriate algorithms for specific priority levels, reducing overall processing time while maintaining data quality.
Solution Approach 2:
The patent implements dynamic algorithm selection by generating models that indicate computing costs for each algorithm and adjusting which algorithms are applied based on the priority of features and available resources. The system can adaptively choose to apply more computationally intensive algorithms to high-priority features while using simpler algorithms or skipping processing for low-priority features.
3Reliability
If comprehensive data cleaning is performed, then data completeness score is improved, but device complexity increases
Solution Approach 1:
The patent performs preliminary action by determining feature priorities and generating algorithm models before actually applying data cleaning algorithms. This pre-processing step creates a structured plan that guides subsequent cleaning operations, reducing the complexity of managing multiple algorithms and data issues during the actual cleaning process.
Solution Approach 2:
The patent incorporates feedback mechanisms by generating models that indicate computing costs and using this information to adjust which algorithms are applied and in what order. The system monitors the cleaning process and can adjust its approach based on the observed data completeness improvements versus computational costs, simplifying the overall system management.
Data Source
AI summary
Methods, systems, and computer program products for prioritized data cleaning are provided herein. A computer-implemented method includes obtaining a dataset comprising a plurality of data issues; determining a priority of one or more features of the dataset; generating a respective model for each of a plurality of data resolution algorithms, wherein each model indicates computing costs of the corresponding data resolution algorithm for resolving at least portion of the plurality of data issues in an order of the priority of the features; and applying one or more of the plurality of data resolutions algorithm to resolve at least a portion of the data issues in the order of the priority of the features based at least in part on the generated models.