Neural Network Classification Throughput via Layer Subsets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Language models face significant computational burdens due to large vocabulary sizes, leading to increased training times and reduced throughput, as existing methods to reduce vocabulary size compromise model capacity and accuracy.
Innovation Solution
A classification apparatus and method using a neural network with a coarse training unit and a fine training unit, where the neural network is trained on subsets of neurons in the last hidden layer and output layer, reducing computational burden while maintaining accuracy and precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the vocabulary set size is increased to improve language model capacity and accuracy, then the model can handle more words and maintain precision, but the computational burden and training time increase significantly
Solution Approach 1:
The patent divides the output layer neurons into multiple subsets, where each subset handles a portion of the vocabulary. During training, different batches are assigned to different subsets, allowing parallel processing and reducing the computational burden per training step while maintaining support for large vocabulary sizes.
Solution Approach 2:
The patent performs preliminary sampling of output layer neurons to create subsets before training. This preliminary action allows the system to pre-organize the computational workload, enabling faster training by avoiding the need to process all output neurons at every training step.
2Adaptability or versatility
If the vocabulary set size is increased to improve language model capacity, then the model can evaluate probabilities for more words, but the computational burden and complexity increase
Solution Approach 1:
The output layer is segmented into multiple subsets of neurons, each responsible for a portion of the vocabulary. This segmentation reduces the computational complexity by allowing parallel processing of different vocabulary portions across multiple training batches, rather than processing the entire vocabulary simultaneously.
Solution Approach 2:
The patent uses partial action by training on subsets of output neurons rather than the complete set at each training step. This approach maintains model capacity for large vocabularies while reducing the immediate computational complexity by processing only a portion of the output layer in each training iteration.
3Productivity
If the number of neurons in the output layer is reduced to decrease computational burden, then training time is reduced, but classification accuracy and precision deteriorate
Solution Approach 1:
The patent dynamically adjusts which output layer neurons are active during different training batches. By cycling through different subsets of output neurons across training batches, the system maintains the ability to support large vocabulary sizes and achieve high classification precision while reducing the computational burden at any given moment.
Solution Approach 2:
The patent ensures continuous coverage of all output layer neurons by systematically cycling through different subsets across training batches. This continuous action ensures that all vocabulary items are eventually processed and contribute to the final model, maintaining classification precision while enabling parallel processing to improve throughput.
4Loss of time
If the vocabulary set is limited to reduce computational time, then training time is reduced by order of 10 times, but model capacity and accuracy are compromised
Solution Approach 1:
The patent segments the output layer into multiple subsets that can be processed in parallel across different training batches. This allows the model to maintain a large vocabulary size (high model capacity) while reducing the computational time per batch, achieving both goals simultaneously rather than requiring a trade-off.
Solution Approach 2:
The patent performs preliminary organization of output neurons into subsets before training begins. This preliminary action enables efficient parallel processing during training, allowing the model to handle large vocabularies without the proportional increase in training time that would otherwise occur, thus preserving model capacity while reducing training time.
Data Source
AI summary
A classification apparatus is configured to perform a classification using a neural network with at least one hidden layer and an output layer, wherein the classification apparatus comprises a coarse training unit configured to train the neural network on a subset of neurons of a last hidden layer and a set of neurons of the output layer; and a fine training unit configured to train the neural network on a set of the last hidden layer and a subset of neurons of the output layer. By executing the classification apparatus, training of a classification model can be improved by reducing the computational burden of the classification, speeding up the training time of a classification model, and speeding up the inference time during application of the classification model.