Cross-Validation Data Subsetting for Time-Series Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current time series forecasting models require significant computational resources and retraining frequency, often resulting in inefficient use of data and potential degradation in model accuracy due to the use of entire datasets for training, especially when data exhibits little to no fluctuation or change.
Innovation Solution
Implement a system that dynamically selects an optimal subset of training data for machine learning models, utilizing a scheduler and cross-validation module to determine necessary retraining based on error thresholds, reducing unnecessary computations and focusing on data that provides value for learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire dataset is used for training the model in each cross-validation fold, then the model training is comprehensive and robust, but the computational resources are significantly consumed and the training process becomes inefficient
Solution Approach 1:
The patent segments the training dataset into multiple subsets (e.g., first subset and second subset) that can be independently evaluated. Instead of training on the entire dataset for each fold, the system trains on one subset, evaluates performance, and determines whether additional subsets are needed based on performance thresholds and data characteristics like stationarity detection.
Solution Approach 2:
The system applies partial action by training on only a portion of the available data when the data exhibits stationarity or when performance thresholds are already met. The cross-validation process dynamically adjusts the amount of data used for training, avoiding unnecessary training on excessive data that would not improve model performance.
2Reliability
If the model is retrained regularly on the entire dataset, then the model stays updated and relevant, but the resource consumption increases significantly and training time is wasted
Solution Approach 1:
The patent implements dynamic retraining decisions based on real-time assessment of data characteristics and model performance. The system monitors whether the time series data exhibits stationarity and whether performance improvements are achieved from additional training, dynamically adjusting the retraining frequency and data scope rather than following a fixed regular schedule.
Solution Approach 2:
The cross-validation process provides feedback on model performance and data characteristics to determine whether retraining is necessary. The system evaluates performance metrics after each training iteration and uses this feedback to decide whether to continue training or stop, avoiding unnecessary retraining cycles that would waste time and resources.
3Measurement precision
If cross-validation is performed on all folds of data, then the model evaluation is thorough and accurate, but the process becomes inefficient when data shows no significant structural changes
Solution Approach 1:
The system performs preliminary analysis of the time series data to detect stationarity and structural changes before conducting full cross-validation. By pre-assessing data characteristics, the system can determine whether comprehensive cross-validation is necessary or whether a simplified evaluation approach would suffice, improving efficiency without sacrificing necessary evaluation accuracy.
Solution Approach 2:
The patent changes the evaluation parameters dynamically based on data characteristics. When data is detected to be stationary or when performance thresholds are met, the system adjusts the cross-validation parameters to reduce the number of folds evaluated or the complexity of evaluation, thereby improving productivity while maintaining adequate measurement precision.
Data Source
AI summary
Provided are systems and methods which optimize a validation process performed during training of a time-series forecasting model. The optimization can remove training data that has poor attributes for training (e.g., less error, less fluctuation, less patterns, etc.) to improve the quality of the training data and reduce the amount of processing that is performed by the host system. In one example, a method may include storing a plurality of machine learning models and a data set, dividing the data set into k folds of data, training the plurality of machine learning models on a subset of folds from among the k folds of data, determining error values for the plurality of machine learning models, respectively, based on fold errors among the subset of folds, and storing the error values within the storage.