Transformer Encoder-Decoder Learning for Dropout Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing online educational systems lack effective methods to predict user drop-out rates and track user knowledge accurately using transformer models, as they are vulnerable to distractions and environmental factors, leading to lower learning efficiency.
Innovation Solution
A transformer structure-based artificial intelligence model optimized for predicting user drop-out rates and tracking user knowledge by using encoder and decoder neural networks, with input data formats that include session position information and response information, and employing attention mechanisms like upper triangular masking to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If transformer models are used to predict user drop-out rates and track user knowledge, then prediction capability is improved, but accuracy is insufficient due to lack of optimized input data formats and prediction methods
Solution Approach 1:
The patent transforms the raw input data into optimized parameters by applying upper triangular masking to the attention mechanism, changing how the transformer model processes sequential data. This parameter transformation enables the model to distinguish between solved and unsolved questions, thereby improving prediction accuracy while maintaining the computational efficiency of the transformer architecture.
Solution Approach 2:
The patent segments the input data into distinct components: question information, response information, and session position information. By dividing the data into these meaningful segments and processing them through separate encoding paths before combining them in the attention mechanism, the system achieves both comprehensive prediction capability and high accuracy.
2Adaptability or versatility
If the system predicts drop-out rates for all questions including unsolved ones, then comprehensive prediction coverage is improved, but prediction accuracy deteriorates due to inclusion of questions that have not yet been attempted
Solution Approach 1:
The patent applies upper triangular masking as a preliminary action before the prediction process. This masking technique pre-processes the attention matrix by setting future (unsolved) question predictions to zero, thereby preventing the model from making inaccurate predictions about questions the user has not yet attempted. This preliminary filtering maintains comprehensive coverage while ensuring accuracy.
Solution Approach 2:
The patent extracts and separates the prediction results for solved versus unsolved questions using the upper triangular masking mechanism. By taking out the inaccurate predictions related to unsolved questions and setting them to zero or excluding them from final evaluation, the system maintains comprehensive prediction coverage while improving overall accuracy metrics.
3Productivity
If conventional RNN models are used for prediction, then implementation simplicity is maintained, but learning speed and performance are insufficient compared to transformer models
Solution Approach 1:
The patent replaces the sequential processing mechanism of RNNs with the parallel attention mechanism of transformers. This substitution eliminates the sequential bottleneck of RNNs where each time step must wait for the previous one, enabling parallel computation across all time steps and thereby achieving faster learning speed while accepting the trade-off of increased model complexity.
Solution Approach 2:
The patent transitions from the temporal sequential dimension of RNNs to the attentional parallel dimension of transformers. By changing the computational dimension from sequential time-step processing to parallel key-query-value attention operations, the system achieves superior learning speed and performance, justified by the enhanced capability rather than simple complexity increase.
Data Source
AI summary
The present disclosure relates to a system capable of increasing the accuracy of a prediction result when predicting a user drop-out rate or predicting a correct answer probability of a user in an online learning environment, and further relates to an operation method of the system. A system according to the present disclosure may include a plurality of encoder neural networks and a plurality of decoders, and may input question information to a kth encoder neural network and input response information to kth decoder neural network to learn the system, thereby predicting the user drop-out rate information and the users correct answer probability information with higher accuracy on the basis of the learned system.