Sparse Sub-Network Training With Divergence Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large neural networks require significant computing resources for training and inference, leading to increased processing time and resource consumption, particularly in online services demanding rapid responses.
Innovation Solution
A training technique for sparsely-activated sub-networks processes batches in two passes, randomly assigning sub-networks for each pass to minimize loss and divergence, eliminating the need for traditional gating mechanisms and ensuring balanced training across sub-networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large neural network is used to improve model performance, then model accuracy is improved, but computing resources and processing time increase
Solution Approach 1:
The neural network is divided into multiple sparsely-activated sub-networks, where each sub-network processes a subset of data. This segmentation allows the system to maintain high model accuracy through multiple specialized sub-networks while reducing processing time by activating only relevant sub-networks for each input batch, rather than processing through the entire large network.
Solution Approach 2:
The system dynamically selects and activates different sub-networks based on the input data characteristics. During training and inference, sub-networks are randomly assigned to process different batches, creating a dynamic processing architecture that adapts to varying data requirements, thereby reducing average processing time while maintaining overall model accuracy.
2Measurement precision
If a large neural network is used to improve model performance, then model accuracy is improved, but computing resources increase
Solution Approach 1:
The large neural network is segmented into multiple smaller sparsely-activated sub-networks. Each sub-network contains a subset of the total parameters, and only the necessary sub-networks are activated for processing each batch. This reduces the immediate computing resource consumption and energy usage while maintaining the overall model accuracy through the collective capability of all sub-networks.
Solution Approach 2:
Different sub-networks are trained with different random assignments of parameters and activations, creating local specialization. Each sub-network develops expertise in specific regions of the parameter space, allowing the system to achieve high overall accuracy while consuming fewer resources by activating only the locally-optimal sub-networks for each input batch.
3Adaptability or versatility
If traditional gating mechanisms are used to route inputs to experts, then input routing is achieved, but algorithmic complexity and load-balancing problems increase
Solution Approach 1:
The traditional gating mechanism is completely removed from the system. Instead of using complex learned gating logic to route inputs to experts, the patent extracts this function and replaces it with simple random assignment of sub-networks to batches. This eliminates the algorithmic complexity of gating mechanisms while maintaining the ability to distribute inputs across multiple sub-networks, and avoids load-balancing problems by using uniform random assignment.
Data Source
AI summary
A training technique trains a neural network having sparsely-activated sub-networks. It does so by processing plural batches of training data in two respective passes of the neural network, yielding first prediction information and second prediction information. For each batch, the technique randomly assigns different sub-networks in the first and second passes of the neural network to process the batch. Over the course of training, the technique attempts to minimize loss information, which describes the difference between the first prediction information and ground-truth information, and the difference between the second prediction information and the ground-truth information. Simultaneously, the technique attempts to minimize divergence information, which describes the divergence of the first prediction information from the second prediction information (and vice versa). The technique can produce an inference-stage model by arbitrarily selecting at least one of the trained sub-networks in the neural network, for use in a production system.