Fast Graph Decoder for Neural Language Model Softmax Approximation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational complexity of the softmax layer in neural language models (NLMs) becomes a bottleneck, especially with large vocabulary sizes, limiting the efficiency of decoding processes in natural language processing tasks and hindering applications in interactive services that require low latency.
Innovation Solution
The Fast Graph Decoder (FGD) approximates the softmax layer by representing the vocabulary as a small world graph, allowing for efficient computation of probabilities for the top-K most likely words, reducing the complexity from linear to logarithmic with respect to the vocabulary size, using Inner Product Preserving Transformation (IPPT) and Hierarchical Navigable Small Worlds (HNSW) for navigation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a softmax layer is used to compute word probabilities in neural language models, then decoding accuracy is maintained, but computational complexity becomes linear with respect to vocabulary size, creating a bottleneck
Solution Approach 1:
The vocabulary is segmented and organized into a hierarchical structure with parent-child relationships. Instead of computing probabilities for all vocabulary words uniformly, the system computes probabilities only for child words of relevant parent nodes, dividing the large vocabulary space into manageable hierarchical segments that reduce computational complexity while maintaining accuracy.
Solution Approach 2:
Parent nodes serve as intermediaries between the context vector and child vocabulary words. The system computes a parent probability distribution first, then uses this intermediate representation to compute child word probabilities, avoiding the need to directly compute probabilities for all vocabulary words and reducing the linear complexity bottleneck.
2Adaptability or versatility
If the vocabulary size is increased to cover more words, then language model coverage improves, but decoding time increases linearly, limiting applications in latency-sensitive services
Solution Approach 1:
The large vocabulary is segmented into hierarchical levels with parent-child relationships. This segmentation allows the system to process only the relevant subset of child words for each parent node during decoding, rather than processing the entire vocabulary, thereby reducing decoding time while maintaining comprehensive vocabulary coverage.
Solution Approach 2:
The hierarchical vocabulary structure is pre-built during training, organizing words into parent-child relationships before decoding occurs. This preliminary organization enables efficient retrieval and probability computation during decoding without requiring linear scanning of the entire vocabulary, thus reducing latency.
3Reliability
If beam search decoder computes probabilities for all vocabulary words, then decoding accuracy is maintained, but processor load and memory usage increase significantly
Solution Approach 1:
The vocabulary is divided into hierarchical segments where only child words of active parent nodes need to be processed. This segmentation dramatically reduces the number of probability computations required at each decoding step, lowering processor load and memory usage while preserving decoding accuracy through the hierarchical structure.
Solution Approach 2:
Instead of computing probabilities for all vocabulary words (excessive action), the system computes probabilities only for the necessary subset of child words associated with active parent nodes in the beam search (partial action). This partial computation approach reduces processor load while maintaining sufficient accuracy for the decoding task.
Data Source
AI summary
Systems, methods, and computer-executable instructions for approximating a softmax layer are disclosed. A small world graph that includes a plurality of nodes is constructed for a vocabulary of a natural language model. A context vector is transformed. The small world graph is searched using the transformed context vector to identify a top-K hypothesis. A distance from the context vector for each of the top-K hypothesis is determined. The distance is transformed to an original inner product space. A softmax distribution is computed for the softmax layer over the inner product space of the top-K hypothesis. The softmax layer is useful for determining a next word in a speech recognition or machine translation.