Training Data Streaming for Unstructured ML Mini-Batch Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large machine learning models require substantial amounts of training data that often do not fit in memory, leading to increased computational burden, resource waste, and inefficiencies in data processing and augmentation, particularly when using data parallelism.
Innovation Solution
A training data stream manager that streams sequenced mini-batches of training data to a training environment, offloading data processing and storage, and enabling efficient generation of training data batches while ensuring reproducibility and reducing resource requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If training data is stored in memory for processing, then data access speed is improved, but memory capacity requirements increase and resource waste occurs when data exceeds memory capacity
Solution Approach 1:
The patent segments training data into mini-batches that are processed sequentially rather than loading entire datasets into memory. The data stream manager divides large training datasets into manageable chunks (mini-batches of 32-256 samples) that fit within available memory, allowing fast processing of each batch while avoiding the need to store entire datasets in memory simultaneously.
Solution Approach 2:
The patent performs preliminary actions by pre-processing and augmenting training data before it needs to be used for model training. The data stream manager pre-loads and prepares mini-batches in advance, applying data augmentations and transformations beforehand so that the model training process can directly consume ready-to-use data without waiting for on-the-fly processing.
2Productivity
If data processing and augmentation are performed during training, then data utilization is improved, but computational burden and processing time increase
Solution Approach 1:
The patent applies data augmentations and transformations in advance during the data loading phase rather than during model training. The data stream manager performs operations such as image resizing, normalization, and other augmentations when data is first loaded into mini-batches, so that the computational burden is shifted away from the critical training loop and does not add to processing time during model optimization.
Solution Approach 2:
The patent introduces a data stream manager as an intermediary component between the training data source and the model training process. This intermediary handles all data processing, augmentation, and mini-batch creation operations, isolating these computational tasks from the model training loop. The stream manager acts as a buffer that prepares data in advance and feeds it to the training process without causing delays.
3Speed
If data parallelism is used for training, then training speed is improved, but resource requirements and system complexity increase
Solution Approach 1:
The patent segments the training data into separate mini-batch streams that can be independently processed and distributed across multiple GPUs or computing nodes. Each processing unit receives its own sequence of mini-batches, allowing parallel training without requiring complex coordination mechanisms. This segmentation enables straightforward data parallelism where each worker processes identical model copies on different data batches.
Solution Approach 2:
The data stream manager serves as a centralized intermediary that manages data distribution to multiple training workers. Instead of having each worker independently manage its own data loading and preprocessing (which would increase complexity), the stream manager centralizes these functions and distributes ready-to-use mini-batches to all workers, simplifying the overall system architecture while enabling efficient data parallelism.
4Quantity of substance
If large datasets are processed in batches, then memory usage is reduced, but I/O operations increase and performance bottlenecks occur
Solution Approach 1:
The patent performs preliminary data loading and caching operations to minimize I/O during training. The data stream manager loads data into memory or cache in advance, organizing it into mini-batch sequences before training begins. This pre-loading reduces the frequency and volume of I/O operations during the actual training process, as data is fetched in larger preliminary chunks rather than in small frequent batches during training.
Solution Approach 2:
The patent ensures continuous data supply to the training process by maintaining a pipeline of pre-prepared mini-batches. The data stream manager continuously generates and queues mini-batches in advance, ensuring that the training process never waits for I/O operations. This continuous action eliminates I/O bottlenecks by decoupling data preparation from data consumption, allowing training to proceed at full speed without interruptions.
Data Source
AI summary
Techniques described herein relate to a method for managing training data. The method includes making a determination that the first stream request is associated with unstructured data; in response to the determination: obtaining a manifest associated with the unstructured data based on the stream request, wherein the manifest comprises metadata; loading the unstructured data into a cache using the manifest; merging the metadata with the unstructured data to generate training data, wherein the training data comprises a plurality of training data examples; generating augmented training data using the training data and the stream specification; generating a mini-batch sequence using the augmented training data and the stream specification; creating a mini-batch sequence queue and a stream endpoint; and streaming the mini-batch sequence using the mini-batch sequence queue and the stream endpoint, wherein the mini-batch sequence is used by a training environment to train a machine learning model.