MapReduce Training Framework for Large-Scale Video Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning techniques are computationally infeasible for training on extremely large datasets, such as those found in video classification tasks, due to the scale and diversity of data, which renders state-of-the-art architectures impractical for deployment at Internet scale.
Innovation Solution
A MapReduce-based training framework that exploits both data parallelism and model parallelism to train machine-learned models, allowing for the optimization of shared feature extraction and prediction heads in parallel using large batches, enabling the use of hundreds of thousands of prediction heads and scaling to datasets 100 times larger than public datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard learning techniques are applied to extremely large datasets (100M+ training examples), then model performance can be improved, but computational feasibility deteriorates
Solution Approach 1:
The patent segments the model into multiple independent components (prediction heads) that can be trained separately in parallel. Each prediction head processes a subset of the training data independently, allowing the system to handle extremely large datasets by dividing the computational workload across multiple processing units without requiring sequential processing of the entire dataset.
Solution Approach 2:
The patent introduces model parallelism as an additional dimension of parallelization beyond data parallelism. By distributing different prediction heads across multiple processing units and enabling simultaneous training of multiple model instances, the system achieves computational scalability that linearly increases with the number of available processing units, making extremely large datasets computationally feasible.
2Adaptability or versatility
If the number of prediction heads is increased to handle diverse video topics, then classification capability is improved, but training time increases
Solution Approach 1:
The patent segments the classification task into multiple independent prediction heads, each responsible for specific classes or topics. This segmentation allows each head to be trained independently and in parallel, so increasing the number of prediction heads to handle more diverse topics does not sequentially increase training time. Instead, the training time scales sub-linearly as new prediction heads can be added and trained simultaneously with existing ones.
3Productivity
If data parallelism is used to scale training, then processing capacity is improved, but communication overhead increases
Solution Approach 1:
The patent segments the model training into independent prediction heads that require minimal inter-communication during training. Each prediction head can be trained independently on different processing units with only occasional synchronization of shared feature extractor parameters, dramatically reducing communication overhead compared to traditional data parallelism where all processing units must frequently synchronize gradients for the entire model.
Data Source
AI summary
A MapReduce-based training framework exploits both data parallelism and model parallelism to scale training of complex models. Particular model architectures facilitate and benefit from use of such training framework. As one example, a machine-learned model can include a shared feature extraction portion configured to receive and process a data input to produce an intermediate feature representation and a plurality of prediction heads that are configured to receive and process the intermediate feature representation to respectively produce a plurality of predictions. For example, the data input can be a video and the plurality of predictions can be a plurality of classifications for content of the video (e.g., relative to a plurality of classes).