Language Model Syntactic Pre-training Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pre-trained language models like BERT lack sufficient syntactic knowledge, leading to syntactically unnatural output when applied to downstream tasks such as phrase extraction, and are prone to catastrophic forgetting during syntactic pre-training.
Innovation Solution
The method involves selecting syntactic pre-training tasks and retraining the pre-trained language model using optimization functions that prevent catastrophic forgetting, allowing the model to retain semantic knowledge while acquiring syntactic knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pre-trained language models are retrained with syntactic pre-training tasks, then syntactic knowledge is improved, but semantic knowledge is lost due to catastrophic forgetting
Solution Approach 1:
The patent applies preliminary action by performing syntactic pre-training tasks before downstream task training. The model is first pre-trained with syntactic knowledge (part-of-speech tagging, dependency parsing, constituency parsing) to establish a strong foundation, then fine-tuned for specific downstream tasks. This preliminary syntactic pre-training prevents catastrophic forgetting of semantic knowledge by establishing syntactic capabilities early in the training pipeline, allowing the model to retain both semantic and syntactic knowledge through the subsequent downstream task training.
2Loss of information
If optimization functions are used to prevent catastrophic forgetting, then retention of semantic knowledge is improved, but training complexity increases
Solution Approach 1:
The patent implements periodic action through a structured training schedule that alternates between syntactic pre-training phases and downstream task fine-tuning phases. During syntactic pre-training, the model learns syntactic patterns from unlabeled text data. Then, during downstream task training, the model applies this syntactic knowledge to specific tasks. This periodic alternation between different training objectives prevents catastrophic forgetting by regularly reinforcing both syntactic and semantic knowledge through structured, alternating training cycles rather than continuous retraining on single tasks.
3Measurement precision
If syntactic pre-training tasks are added to the training pipeline, then syntactic knowledge is enhanced, but training time increases
Solution Approach 1:
The patent applies preliminary action by performing syntactic pre-training tasks before downstream task training. The model is first pre-trained with syntactic knowledge (part-of-speech tagging, dependency parsing, constituency parsing) to establish a strong foundation, then fine-tuned for specific downstream tasks. This preliminary syntactic pre-training prevents catastrophic forgetting of semantic knowledge by establishing syntactic capabilities early in the training pipeline, allowing the model to retain both semantic and syntactic knowledge through the subsequent downstream task training.
Data Source
AI summary
A syntactic pre-training task is selected from among a set of syntactic pre-training tasks. For the selected syntactic pre-training task, the pre-trained language model is retrained by using an optimization function which prevents catastrophic forgetting during the retraining. Inferencing is performed using the retrained language model.