Dataset Caching for Machine Learning Training Trials
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning technologies face inefficiencies in reusing data domain datasets during machine training trials, leading to increased processing overhead and storage requirements due to redundant dataset copies.
Innovation Solution
A computer-implemented method that decodes a parameter code in a computer request for a training trial to identify a next dataset. It checks if the dataset is available in a second computer memory; if available, it supplies that dataset; otherwise, it samples and preprocesses the dataset from a set of sample data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If datasets are repeatedly sampled and preprocessed from sample data for each training trial, then training trials can be executed, but processing overhead increases and storage requirements increase due to redundant dataset copies
Solution Approach 1:
The system preloads datasets into a second computer memory (cache) before they are needed for training trials. By checking availability in advance and loading datasets proactively, the system avoids repeated sampling and preprocessing operations, thereby reducing processing overhead while maintaining training trial execution capability
Solution Approach 2:
The system creates copies of datasets in a second computer memory (cache) that are frequently accessed during training trials. These cached copies allow multiple training trials to access the same dataset without repeatedly reading from the original sample data, reducing redundant processing and storage requirements
2Productivity
If datasets are repeatedly sampled and preprocessed from sample data for each training trial, then training trials can be executed, but storage requirements increase due to redundant dataset copies
Solution Approach 1:
The second computer memory (cache) serves multiple training trials simultaneously by storing shared dataset copies. Instead of each training trial having its own separate dataset copies, the cache provides universal access to the same datasets, reducing total storage requirements while enabling concurrent training trial execution
Solution Approach 2:
The system creates copies of datasets in a second computer memory (cache) that are frequently accessed during training trials. These cached copies allow multiple training trials to access the same dataset without repeatedly reading from the original sample data, reducing redundant processing and storage requirements
3Loss of energy
If the system checks availability of next dataset in second memory before supplying it, then redundant processing is avoided, but additional search operations are required
Solution Approach 1:
The system implements a feedback mechanism where the availability status of datasets in the second memory (cache) is checked before supplying them to training trials. This feedback loop allows the system to determine whether to load from cache or sample from original data, avoiding redundant processing while managing the complexity of availability checking
Data Source
AI summary
A computer product and methodology is provided for supplying data domain datasets in machine training trials. A parameter code is decoded in a computer request for a training trial. The parameter code identifies a next dataset in a set of sample data stored in a first computer memory. The product and methodology further searches to determine whether the next dataset is available in a second computer memory. If the searching determines the next dataset is available in the second memory, then in response to the computer request for a training trial the next dataset that is available in the second memory is supplied. If, on the other hand, the searching determines the next dataset is not available in the second memory, then in response to the computer request for a training trial the next dataset sampled and preprocessed from the set of sample data is supplied.