Shared Encoding Layers for Machine Reading Comprehension
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Fine-tuning pretrained language models for machine reading comprehension tasks often leads to a decrease in generalization performance, particularly when applied to domains outside the training data, due to high dependence on the source domain, which can result in reduced accuracy and increased costs from the need for large amounts of supervised data in the target domain.
Innovation Solution
The approach involves sharing encoding layers between a machine reading comprehension model and a language model through multi-task training, where lower layers are trained jointly and higher layers are divided between the two models, allowing for supervised training on the source domain and unsupervised retraining on the target domain, thereby reducing the need for supervised data in the target domain.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a pretrained language model is fine-tuned for machine reading comprehension task using supervised data, then accuracy in the training domain is improved, but generalization performance in domains not included in training data deteriorates
Solution Approach 1:
The patent segments the encoding layers into two parts: layers 1 to (N-n) that are shared between the language model and machine reading comprehension model, and layers ((N-n)+1) to N that are divided between the two models. This segmentation allows different parts of the model to serve different functions - the shared layers capture general language representations while the divided layers capture task-specific patterns, thereby maintaining both accuracy and generalization performance.
Solution Approach 2:
The patent implements multi-functionality by having the shared encoding layers serve dual purposes: they are used by both the language model for general language understanding and by the machine reading comprehension model for task-specific processing. This universal component captures domain-independent knowledge, enabling the system to maintain good performance across different domains without requiring separate training for each.
2Measurement precision
If a large amount of supervised data is created for machine reading comprehension task, then training accuracy is improved, but creation cost increases
Solution Approach 1:
The patent uses the pretrained language model as a copy or template that already contains valuable language understanding knowledge. Instead of creating all training data from scratch, the system leverages the pretrained model's parameters as a starting point, significantly reducing the amount of new supervised data needed while maintaining high training accuracy.
Solution Approach 2:
The pretrained language model performs preliminary learning of general language patterns and representations before the specific machine reading comprehension task. This preliminary action captures domain-independent knowledge that can be directly transferred to the target task, reducing the need for extensive domain-specific supervised data creation.
3Measurement precision
If a pretrained language model is fine-tuned for specific task, then task-specific accuracy is improved, but dependence on source domain increases
Solution Approach 1:
The patent applies local quality by assigning different functional characteristics to different parts of the model. The shared encoding layers (1 to N-n) have general language understanding properties that are domain-independent, while the divided encoding layers ((N-n)+1 to N) have task-specific properties optimized for machine reading comprehension. This local differentiation allows the model to maintain task-specific accuracy while reducing dependence on the source domain through the general-purpose shared layers.
Data Source
AI summary
An information processing apparatus includes a training unit configured to share encoding layers from a first layer to a (N-n)-th layer having parameters trained in advance by a first model and a second model, and train parameters of a third model through multi-task training including training of the first model and retraining of the second model for a predetermined task, wherein N and n are integers equal to or greater than 1, and satisfies N>n, and in the third model, encoding layers from an ((N-n)+1)-th layer to an N-th layer having parameters trained in advance are divided into the first model and the second model.