Language Generation Model with Multi-Granularity Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current language generation models based on pre-training/fine-tuning frameworks have low accuracy due to relying on semantic representation models that typically use single characters as decoding units, failing to capture the overall semantic meaning of text.
Innovation Solution
A language generation method that performs encoding processing on input sequences to generate hidden state vectors, classifies them to determine granularity categories, and uses multiple decoders to predict language segments at different granularities, updating parameters based on loss values to improve semantic understanding and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single character decoding units are used in language generation models, then the model structure remains simple, but the accuracy of capturing overall semantic meaning deteriorates
Solution Approach 1:
The patent segments the decoding process into multiple decoders, each responsible for different granularity levels (character level, word level, phrase level). This segmentation allows the model to simultaneously maintain structural simplicity for individual decoders while achieving high accuracy in capturing overall semantic meaning through the coordinated work of multiple decoders at different granularities.
2Measurement precision
If multiple decoders are used to predict language segments at different granularities, then the accuracy of language generation improves, but the device complexity increases
Solution Approach 1:
The patent implements a dynamic decoding mechanism where the system can flexibly select and activate appropriate decoders based on the granularity requirements of different language generation tasks. This dynamic approach allows the model to adjust its complexity adaptively, using only the necessary number of decoders for each specific task, thereby improving language generation accuracy without permanently increasing device complexity.
Solution Approach 2:
Each decoder in the patent is designed with multi-functionality, capable of handling different types of language segments (characters, words, phrases) depending on the task requirements. This universality reduces the need for separate specialized decoders for each granularity level, thereby improving language generation accuracy while controlling device complexity through shared computational resources.
3Measurement precision
If encoding processing is performed on input sequences to generate hidden state vectors, then the semantic representation quality improves, but the computational time increases
Solution Approach 1:
The patent performs encoding processing and generates hidden state vectors as a preliminary action before the decoding stage. By completing the encoding process upfront and storing the hidden state vectors, the system prepares high-quality semantic representations in advance, which can then be efficiently utilized by multiple decoders without repeatedly performing computationally intensive encoding operations, thus improving semantic representation quality while managing computational time.
Data Source
AI summary
The present disclosure proposes a language generation method and apparatus. The method includes: performing encoding processing on an input sequence by using a preset encoder to generate a hidden state vector corresponding to the input sequence; in response to a granularity category of a second target segment being a phrase, decoding a first target segment vector, the hidden state vector, and a position vector corresponding to the second target segment by using N decoders to generate N second target segments; determining a loss value based on differences between respective N second target segments and a second target annotated segment; and performing parameter updating on the preset encoder, a preset classifier, and the N decoders based on the loss value to generate an updated language generation model for performing language generation.