Dynamic Training Data Partitioning for Parallel Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional distributed algorithms for parallel training of machine learning models face challenges in achieving significant acceleration due to unfavorable convergence rate dependence on the number of workers, leading to increased epochs required for convergence, especially when scaling out to multiple cores.
Innovation Solution
Dynamic partitioning of training data across workers by shuffling or offsetting subsets, allowing different subsets to be used over time, which can be reshuffled within or across epochs, improving convergence rate and scalability by reducing reliance on shared memory access and enabling efficient utilization of multi-threaded architectures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional distributed algorithms are used for parallel training, then training can be performed on multiple workers, but the convergence rate deteriorates with increasing number of workers
Solution Approach 1:
The patent implements dynamic data partitioning where the assignment of training data subsets to workers changes over time. Workers dynamically receive different data partitions in different iterations, allowing the system to adapt to convergence requirements while maintaining parallel execution. This dynamic approach prevents the convergence degradation seen in static partitioning schemes when scaling to many workers.
Solution Approach 2:
The training data is segmented into multiple subsets that are distributed across workers. Each worker processes a specific subset, and the segmentation is dynamically reassigned to ensure optimal convergence. This segmentation allows parallel processing while the dynamic reassignment maintains effective utilization of all workers throughout training.
2Device complexity
If static partitioning of training data is used across workers, then data distribution is simple, but convergence rate is slow and epochs increase
Solution Approach 1:
The system transitions from static to dynamic data partitioning. The data assignment to workers is no longer fixed but changes dynamically during training, allowing the system to maintain fast convergence rates even as the number of workers increases. This dynamic reassignment is the key factor in reducing epoch requirements compared to static partitioning.
3Productivity
If more workers are added to accelerate training, then processing capacity increases, but convergence rate dependence becomes unfavorable
Solution Approach 1:
The dynamic data partitioning strategy allows the system to maintain stable convergence rates regardless of the number of workers. By continuously adapting data assignments, the system ensures that each worker contributes effectively to convergence, preventing the unfavorable dependence on worker count that plagues traditional distributed algorithms.
Data Source
AI summary
Parallel training of a machine learning model on a computerized system is described. Computing tasks of a system can be assigned to multiple workers of the system. Training data can be accessed. The machine learning model is trained, whereby the training data accessed are dynamically partitioned across the workers of the system by shuffling subsets of the training data through the workers. As a result, different subsets of the training data are used by the workers over time as training proceeds. Related computerized systems and computer program products are also provided.