Data Quality Assessment via Feature Reconstruction Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Identifying and addressing data quality issues in large datasets with numerous features is challenging, as pinpointing abnormal observations and determining their causes requires individual review of each feature, and existing methods fail to efficiently determine the impact of other features on data quality.
Innovation Solution
A model quality system uses data reconstruction models to generate machine learning models for each feature, predicting values based on other features, and calculates feature impact parameters to identify sources of data quality issues by comparing predicted and observed values, thereby determining the relative impact of other features and transmitting data inspection requests to their sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If individual review of each feature is performed to determine data quality issues, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The system segments the data quality assessment task by creating separate machine learning models for each feature, where each model independently predicts that specific feature's values. This allows parallel processing of multiple features simultaneously rather than sequential individual review, maintaining comprehensive assessment precision while reducing total time required.
Solution Approach 2:
The system employs self-service by using machine learning models to automatically assess data quality without requiring manual review. The models self-evaluate each feature against predicted values generated from other features, automatically identifying anomalies and determining data quality issues without human intervention in the assessment process.
2Productivity
If machine learning models are generated for each feature to predict values, then productivity is improved, but device complexity increases
Solution Approach 1:
The system divides the complex data quality assessment problem into separate, manageable machine learning models, with each model dedicated to predicting a specific feature. This segmentation allows each model to be trained and optimized independently, improving overall productivity while keeping individual model complexity manageable through modular architecture.
Solution Approach 2:
The system creates a universal framework where machine learning models serve multiple functions: predicting feature values, identifying data quality issues, determining feature impacts, and guiding inspection requests. This multi-functionality consolidates what would otherwise require separate systems into a single integrated platform, improving productivity without proportionally increasing complexity.
3Measurement precision
If feature impact parameters are calculated to identify sources of errors, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The system performs preliminary action by pre-calculating feature impact parameters during the machine learning model training process. These impact parameters are determined in advance based on the relationships between features, allowing rapid identification of error sources without requiring time-consuming post-hoc analysis when data quality issues are detected.
Solution Approach 2:
The system uses feedback mechanisms where predicted values from machine learning models are compared against actual observed values to automatically identify data quality issues. The feature impact parameters provide feedback on which features contribute most to prediction errors, enabling precise error source identification through automated feedback loops rather than manual analysis.
Data Source
AI summary
Methods and systems are described herein for determining data quality using data reconstruction models. The system receives a dataset including entries and features and generates a machine learning model for each feature of the dataset. Each model may be trained to generate predictions for a corresponding feature based on other features of the dataset. The system may input, into each model, values of the other features to obtain prediction values for the corresponding feature. For a subset of entries for which a difference between the predicted and actual values of the corresponding feature satisfies a threshold, the system may determine relative impacts of the other features on the corresponding feature. The system may then transmit, to a user, a subset of the other features having relative impacts that meet a feature impact threshold.