Continuous Embedding Output Layer for Neural Network Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural machine translation models are computationally expensive and parameter-heavy due to their reliance on one-hot representations and softmax functions for word outputs, leading to inefficient decoding processes.
Innovation Solution
A machine learning system with a continuous embedding output layer that generates continuous vectors, utilizing a cosine similarity-based loss function to train the model, replacing the softmax layer with a continuous valued embedding table to output word IDs as nearest neighbors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If softmax function with one-hot representations is used for word output, then language generation accuracy is maintained, but computational complexity and parameter count increase significantly
Solution Approach 1:
The patent changes the output parameter representation from discrete one-hot vectors to continuous vectors. The continuous embedding layer outputs a continuous vector that is then converted to discrete word IDs through nearest neighbor search in the embedding table, replacing the traditional softmax computation over the entire vocabulary.
Solution Approach 2:
The patent extracts only the necessary directional information from the full softmax computation. By using continuous vectors and cosine similarity measurement, the system extracts the essential directional relationship between output and target embeddings without computing probabilities for all vocabulary items.
2Reliability
If softmax layer is used for decoding, then probability distribution over vocabulary is obtained, but decoding speed decreases due to large matrix multiplication
Solution Approach 1:
The patent introduces continuous vectors as an intermediary representation between the neural network output and the discrete word vocabulary. The continuous embedding layer produces continuous vectors that serve as intermediaries, which are then efficiently mapped to word IDs through nearest neighbor search, avoiding direct softmax computation over the vocabulary.
Solution Approach 2:
The patent replaces the mechanical softmax computation (large matrix multiplication) with a different mechanism: continuous vector generation followed by nearest neighbor search in embedding space. This substitution uses cosine similarity measurement and continuous embedding lookup instead of probabilistic softmax computation.
3Measurement precision
If one-hot representations are used for each word, then precise word identification is achieved, but parameter count and memory usage increase
Solution Approach 1:
The patent changes the parameter representation from high-dimensional one-hot vectors to low-dimensional continuous vectors. The continuous embedding layer outputs continuous vectors with fewer dimensions than the vocabulary size, and the embedding table stores dense continuous representations instead of one-hot encodings.
Solution Approach 2:
The patent transitions from the discrete high-dimensional one-hot space to a continuous lower-dimensional embedding space. By representing words as continuous vectors in a compressed dimensional space, the system maintains semantic relationships while reducing the number of parameters required.
Data Source
AI summary
A machine learning system including a continuous embedding output layer is provided. Whereas traditional machine language translation or generation models utilize an output layer that include an single output for each word in the output vocabulary V, the present machine learning system includes a continuous embedding output layer that stores continuous vectors mapped to an m-dimensional vector space, where m is less than V. Accordingly, the present machine learning system processes an input string to produce an output vector and then searches for the continuous vector within the vector space that most closely corresponding to the output vector via, for example, a k-nearest neighbor algorithm. The system then outputs the output string corresponding to the determined continuous vector. The present system can be trained utilizing a cosine-based loss function.