NLP Model Knowledge Retention via Segmented Embedding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing models face challenges in retaining information from external knowledge as input embedding vectors are transformed during inference operations, leading to increased parameter and computation requirements due to larger embedding vector sizes.
Innovation Solution
A processor-implemented method that performs word embedding, encoding, and decoding using a natural language processing model to generate context embedding vectors, calculates scores for true and false knowledge embedding vectors, and trains the model based on losses derived from these scores, incorporating knowledge information through a knowledge graph embedding process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the size of the embedding vector is increased to include more knowledge information, then the knowledge representation capability is improved, but the parameter quantity and computation quantity of the natural language processing model increase
Solution Approach 1:
The patent segments the knowledge integration process into two distinct components: (1) word embedding layer that generates initial embeddings with knowledge information, and (2) encoder layer that processes these embeddings to generate context embeddings. This segmentation allows knowledge information to be injected at the word embedding level without requiring the entire model to scale proportionally, thus improving knowledge retention while controlling overall model complexity.
Solution Approach 2:
The patent applies preliminary action by pre-processing knowledge information into embedding vectors before they are fed into the main natural language processing model. The word embedding layer performs this preliminary transformation, converting external knowledge into a format that can be efficiently processed by the encoder, thereby reducing the computational burden on the main model while ensuring knowledge information is preserved from the outset.
2Loss of information
If the size of the embedding vector is increased to include more knowledge information, then the knowledge representation capability is improved, but the computation quantity of the natural language processing model increases
Solution Approach 1:
The computation process is segmented into distinct stages: word embedding computation (which handles knowledge information) and encoder computation (which handles language processing). This segmentation allows the model to efficiently process knowledge information at the embedding level without requiring proportional increases in the computational resources of the entire model, thus reducing overall computation quantity while maintaining knowledge retention.
Solution Approach 2:
Knowledge information is pre-processed into embedding vectors before entering the main computation pipeline. This preliminary transformation consolidates knowledge information into compact representations that require fewer computational operations during the main encoding and decoding processes, thereby reducing the overall computation quantity and energy consumption of the natural language processing model.
3Device complexity
If traditional natural language processing models are used without knowledge graph integration, then the model complexity is kept simple, but the knowledge information retention during inference operations is lost
Solution Approach 1:
The patent merges the knowledge graph embedding functionality with the traditional natural language processing model by integrating the word embedding layer that incorporates external knowledge. This merging allows the model to retain knowledge information throughout the inference process while maintaining a relatively simple overall structure, as the knowledge integration is achieved through the embedding layer rather than through complex architectural modifications.
Data Source
AI summary
A method and device with natural language processing is disclosed. The method includes performing a word embedding of an input sentence, encoding a result of the word embedding, using an encoder of a natural language processing model, to generate a context embedding vector, decoding the context embedding vector, using a decoder of the natural language processing model, to generate an output sentence corresponding to the input sentence, generating a score indicating a relationship between the context embedding vector and each of a plurality of knowledge embedding vectors, determining a first loss based on the output sentence, determining a second loss based on the generated score, and performing training of the natural language processing model, including training the natural language processing model based on the determined first loss, and training the natural language processing model based on the determined second loss.