Span Selection Pretraining for BERT Semantic Completion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language processing systems based on BERT models, while effective, can be improved for tasks like question-and-answer systems, word prediction, and next sentence prediction, as they do not fully leverage the potential of bidirectional encoder representations for certain semantic completion tasks.
Innovation Solution
The implementation of span selection pretraining, which involves generating training data by masking a word in a sentence and replacing it with a placeholder, and selecting the highest-ranked passage that includes the answer term, to enhance the BERT model's performance by capturing dependencies across the entire sequence rather than just left-to-right or right-to-left processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard BERT pretraining is used with left-to-right or right-to-left processing, then the model can process sequences efficiently, but it cannot fully capture bidirectional dependencies for semantic completion tasks
Solution Approach 1:
The patent applies preliminary action by implementing span selection pretraining before fine-tuning on specific tasks. The BERT model is first pretrained on a span selection task where masked spans must be predicted from candidate passages, establishing strong bidirectional representations in advance. This preliminary pretraining phase captures contextual dependencies more effectively than standard approaches, improving downstream task performance without adding complexity during the actual task execution phase.
2Measurement precision
If span selection pretraining is implemented with masked words and candidate passages, then the model captures dependencies across the entire sequence, but the training data generation and processing becomes more complex
Solution Approach 1:
The patent applies copying by creating multiple candidate passages for each masked span in the training data. Instead of using the original passage directly, the system generates candidate spans (including correct and incorrect options) and creates training examples where the model must select the correct span from these copied variations. This copying approach enables the model to learn discriminative features for span selection, improving exact match accuracy while keeping the training framework manageable through systematic candidate generation.
Data Source
AI summary
Methods and systems for natural language processing include pretraining a machine learning model that is based on a bidirectional encoder representations from transformers model, using a span selection training data set that associates a masked word with a passage. A natural language processing task is performed using the span selection pretrained machine learning model.