ML Pipeline Test Data Generation via Master Set Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning (ML) systems are sensitive to the quality of their training data sets, and poor data quality can lead to ML models that are either ineffective or misleading, wasting computational resources and time.
Innovation Solution
An ML trainer pipeline that prepares and analyzes training data before building an ML model, using a sophisticated filtering technique to select test data sets from a master data set, ensuring that only high-quality data sets are used for model building, thereby avoiding the construction of unusable models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If test data sets are synthetically generated for ML pipeline testing, then testing coverage can be achieved, but the quality and representativeness of the data is poor and the data does not accurately reflect real-world distributions
Solution Approach 1:
The patent copies real production data into test data sets instead of synthetically generating data. A data copying module retrieves actual data from production databases and transforms it into test data sets, ensuring the data accurately reflects real-world distributions and characteristics while avoiding the quality issues of synthetic data generation.
2Reliability
If large master data sets are retained for test data generation, then high-quality test data can be generated, but storage requirements increase significantly
Solution Approach 1:
The patent extracts only the necessary portions of production data needed for testing rather than retaining entire master data sets. The system copies and transforms specific data subsets into test data sets on-demand, using minimal storage space while maintaining data quality through selective extraction from production databases.
Solution Approach 2:
The patent implements dynamic test data generation where test data sets are created on-demand from production data rather than pre-storing all possible test data. This dynamic approach allows the system to generate high-quality test data only when needed, optimizing the balance between data quality and storage requirements.
3Productivity
If ML models are trained with poor quality data sets, then training time is wasted, but computational resources are consumed unnecessarily
Solution Approach 1:
The patent performs preliminary quality assessment of training data sets before ML model training begins. The system evaluates data quality metrics and determines whether data meets minimum thresholds for effective training, preventing wasteful consumption of computational resources on poor quality data sets before the expensive training process starts.
Data Source
AI summary
A system may include memory containing: (i) a master data set representable in columns and rows, and (ii) a query expression. The system may include a software application configured to apply a machine learning (ML) pipeline to an input data set. The system may include a computing device configured to: obtain the master data set and the query expression; apply the query expression to the master data set to generate a test data set, where applying the query expression comprises, based on content of the query expression, generating the test data set to have one or more columns or one or more rows fewer than the master data set; apply the ML pipeline to the test data set, where applying the ML pipeline results in either generation of a test ML model from the test data set or indication of an error in the test data set; and delete the test data set from the memory.