Distributed Neural Network Training via Exploration-Exploitation Phases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing synchronous and asynchronous methods for distributed training of neural networks are communication-intensive and result in a generalization gap, leading to slower training and poor performance on unseen data due to large mini-batch sizes.
Innovation Solution
The proposed method employs an exploration-exploitation phase approach, where an exploration phase uses local optimization with reduced communication and an exploitation phase utilizes larger mini-batches for global information merging, addressing communication intensity and generalization gaps by seamlessly integrating internal state transmission between phases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If synchronous or asynchronous distributed training methods are used, then training can be performed across multiple computing devices, but communication overhead increases and generalization performance deteriorates
Solution Approach 1:
The training process is segmented into two distinct phases: exploration phase where each computing device trains independently with local data subsets, and exploitation phase where models are aggregated and refined collectively. This segmentation reduces communication frequency and overhead while maintaining distributed training benefits.
Solution Approach 2:
A parameter server acts as an intermediary that collects model parameters from multiple computing devices during the exploitation phase, aggregates them, and redistributes updated parameters. This mediator approach streamlines communication and reduces direct peer-to-peer communication overhead.
2Loss of time
If large mini-batch sizes are used in distributed training, then communication frequency decreases, but generalization gap increases and performance on unseen data deteriorates
Solution Approach 1:
The system dynamically adjusts mini-batch sizes and communication strategies based on the training phase. During exploration phase, smaller batches are used with reduced communication, while during exploitation phase, larger batches are employed for model aggregation, optimizing both training efficiency and generalization performance.
Solution Approach 2:
Communication between computing devices and the parameter server occurs periodically at phase boundaries rather than continuously. The exploration phase performs multiple local updates with minimal communication, followed by periodic exploitation phase aggregations, creating a rhythmic communication pattern that reduces overall overhead.
3Loss of energy
If exploration phase with local optimization is used, then communication overhead is reduced, but convergence speed may slow without global information merging
Solution Approach 1:
The exploration phase performs preliminary local optimization on each computing device using local data subsets before the exploitation phase. This preliminary action prepares initialized models that are closer to optimal solutions, reducing the work needed during global aggregation and accelerating overall convergence.
Solution Approach 2:
The exploitation phase merges models from multiple computing devices through parameter aggregation on the parameter server. This combining of locally optimized models leverages diverse local optimizations to achieve faster global convergence than any single device could achieve alone.
4Speed
If exploitation phase with global information merging is used, then convergence speed improves, but communication intensity increases
Solution Approach 1:
The exploitation phase extracts and aggregates only the essential model parameters from each computing device rather than transmitting complete model states or intermediate computations. This extraction approach reduces communication intensity while maintaining the benefits of global information merging for convergence.
Data Source
AI summary
A computing system provides distributed training of a neural network model. Explore phase options, exploit phase options, a subset of a training dataset, and a validation dataset are distributed to a plurality of computing devices. (a) Execution of the model by the computing devices is requested using the subset stored at each computing device. (b) A first result of the execution is received from a computing device. (c) Next configuration data for the neural network model is selected based on the first result and distributed to the computing device. (a) to (c) is repeated until an exploration phase is complete. (d) Execution of the neural network model is requested. (e) A second result is received. (f) Next configuration data is computed based on the second result and distributed to the computing device. (d) to (f) is repeated until an exploitation phase is complete. The next configuration data defines the model.