Abstract Description Model Training via Backward-Derived Hidden States
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional abstract description models suffer from low prediction accuracy due to the lack of consideration for the correlation between neighboring hidden states during the training process, leading to a difference between training and prediction processes.
Innovation Solution
The proposed method involves a two-phase training process for the abstract description model, where the first phase uses a recurrent neural network to obtain hidden states and a first loss function for maximum likelihood estimation, and the second phase incorporates backward derivation to obtain backward-derived hidden states and a second loss function to account for the correlation between neighboring hidden states, adjusting model parameters based on the second loss function's preset threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the decoding network independently predicts words using only the current hidden state, then the prediction process is simple and fast, but the prediction accuracy is low due to ignoring context correlation
Solution Approach 1:
The patent introduces feedback mechanisms by computing backward-derived hidden states from future time steps and incorporating them into the current prediction. The backward-derived hidden state h̃t-1 is computed from ht+1 and used to supervise the current hidden state ht, creating a feedback loop that enforces temporal consistency and improves prediction accuracy by considering future context.
Solution Approach 2:
The patent performs preliminary computation of backward-derived hidden states during the training phase. By pre-computing h̃t-1 from future observations and using it to supervise current predictions, the model prepares contextual information in advance, allowing the decoding network to make more accurate predictions without adding significant online computational complexity.
2Measurement precision
If the model uses backward derivation to consider correlation between neighboring hidden states, then the prediction accuracy improves, but the training process becomes more complex with additional loss functions
Solution Approach 1:
The patent segments the training process into two distinct phases: the first phase trains the model using traditional forward prediction with cross-entropy loss, and the second phase refines the model using backward derivation with additional loss terms. This segmentation allows the model to first learn basic prediction capabilities and then progressively incorporate temporal consistency constraints, making the complex training process more manageable and effective.
Solution Approach 2:
The patent implements dynamic training by adaptively adjusting the loss function composition across training phases. The training dynamics evolve from simple cross-entropy minimization to a more complex objective that includes backward-derived state consistency terms, allowing the model to progressively learn temporal relationships without overwhelming the optimization process.
3Reliability
If the model considers correlation between neighboring hidden states through backward derivation, then the difference between training and prediction is reduced, but the computational overhead increases
Solution Approach 1:
The patent performs preliminary computation of backward-derived hidden states during the training phase, pre-computing h̃t-1 from future observations. This preliminary action ensures that the training process accurately reflects the prediction process, reducing the training-prediction gap while the computational overhead is concentrated during the offline training phase rather than affecting online prediction speed.
Data Source
AI summary
This application relates to an abstract description generating method, an abstract description generation model training method, a computer device, and a storage medium. The abstract description generating method includes: inputting a labeled training sample into an abstract description generation model; performing first-phase training on an encoding network and a decoding network of the abstract description generation model based on supervision of a first loss function; obtaining a backward-derived hidden state of a previous moment through backward derivation according to a hidden state of each moment outputted by the decoding network; obtaining a value of a second loss function according to the backward-derived hidden state of the previous moment and an actual hidden state of the previous moment outputted by the decoding network; and obtaining final model parameters of the abstract description generation model determined based on supervision of the second loss function to reach a preset threshold value.