Data Quality Framework for ML Pipeline Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models often produce unstable and marginal improvements due to poor quality data, leading to frequent re-training and increased costs, as there is a knowledge gap between data experts and ML experts, resulting in the 'garbage-in, garbage-out' problem.
Innovation Solution
A Data Quality Framework (DQF) is implemented within the machine learning pipeline to autonomously assess and improve the quality of input data by generating a data quality score using ensemble statistical techniques, filtering out poor data, and recommending corrective actions until the data meets a specified threshold, ensuring high-quality data for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If all available data is used for ML training, then the quantity of training data is maximized, but the quality of training data deteriorates leading to garbage-in garbage-out problems
Solution Approach 1:
The patent segments the training data into high-quality and low-quality portions using automated quality assessment. The data quality framework evaluates each data point against defined criteria and separates acceptable data from unacceptable data, allowing the ML system to train primarily on high-quality data while maintaining sufficient data quantity for effective training.
Solution Approach 2:
The patent applies preliminary data quality assessment and filtering before the ML training process begins. The data quality framework pre-evaluates all candidate training data, identifies quality issues, and prepares a cleaned dataset in advance, preventing garbage data from entering the training pipeline and eliminating the need for post-training corrections.
2Measurement precision
If ML model tuning is performed rigorously to improve results from poor quality data, then prediction accuracy may be marginally improved, but the cost and time required increases significantly
Solution Approach 1:
The patent performs data quality assessment and correction as a preliminary step before ML model training and tuning. By addressing data quality issues upfront through automated evaluation and filtering, the system eliminates the need for extensive subsequent model tuning, reducing both time and computational costs while maintaining prediction accuracy.
Solution Approach 2:
The patent converts the harmful effect of poor data quality into a beneficial filtering process. The data quality framework identifies data deficiencies and automatically applies corrections or exclusions, transforming what would be a source of error into an opportunity to improve overall system performance and reduce tuning requirements.
3Productivity
If ML models are trained with poor quality data, then training can proceed without data filtering, but the models become unstable and require frequent re-training
Solution Approach 1:
The patent implements preliminary data quality filtering before training to ensure model stability. The data quality framework assesses and cleans the training data in advance, preventing unstable models from being trained in the first place. This approach maintains training efficiency by avoiding the need for frequent re-training while ensuring long-term model stability.
Solution Approach 2:
The patent incorporates feedback mechanisms where the data quality framework continuously monitors data quality metrics and adjusts filtering criteria based on model performance. This feedback loop ensures that data quality standards are maintained over time, preventing model degradation and reducing the frequency of re-training while preserving training efficiency.
Data Source
AI summary
Systems, methods, and other embodiments associated with a data quality framework for a machine learning pipeline are described. In one embodiment, a method includes receiving an input dataset prior to training a machine learning model with the input dataset. The input dataset, having data records in column tabular form, is analyzed to determine a data quality of the input dataset prior to training the machine learning model. A data quality score is generated that represents an overall quality of the input dataset, where the data quality score is generated based on a combined ensemble of at least two factors selected from (i) an abundance factor, (ii) a completeness factor, and (iii) a dimension efficiency factor. Based on the data quality score, the input dataset is either permitted or prohibited from continuing in the machine learning pipeline, and corrective actions may be performed.