Softmax Approximation via Vocabulary Clustering for Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks with large vocabularies face computational bottlenecks in inference times due to the softmax layer, which can delay implementation on resource-constrained devices like mobile devices.
Innovation Solution
Implementing a lightweight screening model that approximates the softmax layer by predicting a proper subset of the vocabulary, reducing the number of inner products required for computation, thereby decreasing inference time and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a conventional softmax output layer is used to compute probability distribution over large vocabulary, then accurate output probabilities are obtained, but inference time increases significantly
Solution Approach 1:
The patent segments the vocabulary into multiple clusters, with each cluster having an associated cluster vector. Instead of computing softmax over the entire vocabulary, the system computes softmax over clustered representations, significantly reducing the computational space while maintaining accuracy through the clustering structure.
Solution Approach 2:
The patent transforms the problem by changing parameters from individual vocabulary item representations to cluster-level representations. By computing inner products between context vectors and cluster vectors rather than all vocabulary vectors, the system reduces computational complexity while preserving essential information through the clustering parameter transformation.
2Reliability
If softmax computation is performed over the entire vocabulary, then complete probability distribution is generated, but computational resources increase
Solution Approach 1:
The vocabulary is segmented into multiple clusters, allowing the system to process grouped representations rather than individual items. This segmentation reduces the number of computations required while maintaining reliability through the structured organization of vocabulary items into meaningful clusters.
Solution Approach 2:
The system performs partial action by computing probabilities only over cluster representatives rather than all vocabulary items. This partial computation is sufficient to maintain reliable probability distributions while significantly reducing computational resource requirements.
3Adaptability or versatility
If the vocabulary size is large, then more output categories are covered, but the number of inner products required increases
Solution Approach 1:
The large vocabulary is segmented into multiple clusters, each represented by a cluster vector. This segmentation allows the system to maintain comprehensive vocabulary coverage while reducing the number of inner product computations by operating on cluster-level representations rather than individual vocabulary items.
Solution Approach 2:
Each cluster vector serves as a universal representation for multiple vocabulary items within that cluster. This multi-functionality allows a single cluster vector to represent many individual vocabulary items, reducing the total number of computations needed while maintaining comprehensive vocabulary coverage.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for decreasing neural network inference times using softmax approximation. One of the methods includes maintaining data specifying a respective softmax weight vector for each output in a vocabulary of possible neural network outputs; receiving a neural network input; processing the neural network input using one or more initial neural network layers to generate a context vector for the neural network input; and generating an approximate score distribution over the vocabulary of possible neural network outputs for the neural network input, comprising: processing the context vector using a screening model configured to predict a proper subset of the vocabulary for the context input; and generating a respective logit for each output that is in the proper subset, comprising applying the softmax weight vector for the output to the context vector.