Continuous Machine Learning System for Limited Resource Containers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning training systems require sufficient resources to handle entire data pools, which is not feasible in containerized environments with limited memory, leading to incomplete training or system crashes.
Innovation Solution
A continuous machine learning system comprising a data generator module, pipeline search module, and pipeline refinement module that generates data batches, determines the best machine learning model pipeline, refines it, and incrementally trains it using subsequent data batches, allowing training in environments with limited resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional machine learning training systems are used to handle entire data pools, then training completeness can be achieved, but system resources (memory) are insufficient in containerized environments
Solution Approach 1:
The patent divides the entire training data pool into multiple smaller batches that can be processed sequentially. The data generator module creates data batches with controlled sizes, allowing the system to train on portions of data that fit within available memory constraints while ultimately processing the complete dataset through multiple passes.
Solution Approach 2:
The system performs partial training actions on data batches rather than attempting to load the entire dataset at once. By training incrementally on subsets of data and iterating through multiple batches, the system achieves complete training coverage without requiring memory resources proportional to the full dataset size.
2Quantity of substance
If data is broken into batches for limited resource environments, then resource utilization improves, but training time increases
Solution Approach 1:
The data generator module pre-processes and organizes the training data into batches before training begins. This preliminary organization allows the training process to proceed efficiently through pre-segmented data without the overhead of dynamic data loading or processing during training iterations.
Solution Approach 2:
The system maintains continuous training action by seamlessly processing through multiple data batches without interruption. The pipeline modules are designed to handle batch transitions efficiently, ensuring that the training process continues without significant idle time between batches, thereby minimizing total training time despite processing data in segments.
3Speed
If the entire data pool is loaded into memory, then training speed can be maximized, but system crashes occur in containerized environments with limited memory
Solution Approach 1:
By segmenting the data pool into smaller batches that fit within available memory, the system avoids the memory overflow that causes system crashes. Each batch is processed independently within memory constraints, ensuring system stability while maintaining training progress.
Solution Approach 2:
The data generator module acts as an intermediary between the storage system and the training pipeline, managing data flow in controlled batches. This intermediary layer prevents direct loading of the entire dataset into memory, thereby preventing system crashes while enabling continuous training through batched data supply.
Data Source
AI summary
A continuous machine learning system includes a data generator module, a pipeline search module, a pipeline refinement module, and a pipeline training module. The data generator module obtains raw training data defining a total data size and generates a plurality of data batches from the raw training data. The pipeline search module obtains an initial data batch from among the plurality of data batches and determines a best machine learning model pipeline among a plurality of machine learning model pipelines based on the initial data batch. The pipeline refinement module receives the best machine learning model pipeline and refines the best machine learning model pipeline to generate a refined pipeline that consumes the plurality of data batches. The pipeline training module incrementally trains the refined pipeline using remaining data batches among the plurality of data batches generated after the initial data batch.