AI/ML Training Data Reduction Through Redundant Variable Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern AI/ML models require large amounts of training data, which can be difficult, time-consuming, and expensive to obtain, and existing methods for reducing this burden are often suboptimal or require significant computational resources.
Innovation Solution
Identify redundant variables in observation information using techniques such as the two-sample Kolmogorov-Smirnov test to determine if variables come from the same probability distribution, allowing for the reduction of training data by excluding redundant observations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large amounts of training data are used to train modern AI/ML models, then model performance and accuracy are improved, but data collection becomes more time-consuming and expensive
Solution Approach 1:
The patent extracts and removes redundant variables from the training data by identifying variables that provide no additional information beyond what is already captured by other variables. This extraction process reduces the training data size while preserving the essential information needed for model training, thereby reducing data collection time without compromising model performance
Solution Approach 2:
The patent changes the parameter space by transforming the training data from a high-dimensional space with many variables to a reduced-dimensional space with fewer variables. This parameter transformation is achieved through statistical analysis that identifies and eliminates redundant variables, effectively changing the structure of the training data while maintaining model training effectiveness
2Reliability
If large amounts of training data are used to train modern AI/ML models, then model performance and accuracy are improved, but computational resources and training costs increase
Solution Approach 1:
The patent extracts redundant variables from the training data, reducing the amount of data that needs to be processed during training. This extraction directly reduces computational resource requirements while maintaining the essential information needed for model performance, thereby lowering training costs without sacrificing model quality
Solution Approach 2:
The patent transforms the training data parameters by reducing the number of variables from the original high-dimensional space to a reduced-dimensional space. This parameter change decreases the computational complexity of training while preserving the predictive power of the model, effectively reducing energy consumption and training costs
3Productivity
If redundant variables are removed from training data, then training time and computational resources are reduced, but data completeness may be compromised
Solution Approach 1:
The patent employs statistical analysis and testing mechanisms that provide feedback to verify whether variables are truly redundant before removing them. This feedback loop ensures that only variables that do not contribute to model performance are removed, thereby maintaining data completeness and preventing information loss while still achieving faster training speeds
Solution Approach 2:
The patent replaces mechanical data processing with statistical analysis and mathematical testing to determine variable redundancy. This substitution uses rigorous statistical methods to verify that removed variables are truly redundant, ensuring that data completeness is maintained while enabling faster training through reduced data size
Data Source
AI summary
A method includes obtaining observation information related to an artificial intelligence/machine learning (AI/ML) model to be trained and identifying multiple variables associated with the observation information. The method also includes analyzing at least a portion of the observation information associated with the identified variables to determine whether the identified variables are redundant and determining that two or more of the identified variables are redundant with one another based on the analysis. The method further includes obtaining a set of training data for training the AI/ML model, where the set of training data includes observations over a range of values for at least one of the two or more variables determined to be redundant and lacks observations over a range of values for at least one other of the two or more variables determined to be redundant.