Semi-Supervised Learning Framework for Encoder Model Pre-Training

The patent applies preliminary action by performing pre-training on unlabeled data before supervised fine-tuning on labeled data. This preparatory step allows the model to learn general features and patterns from abundant unlabeled data, so that when supervised training occurs later, it requires less time and fewer resources to achieve the same level of accuracy.

The patent implements continuity of useful action by maintaining a two-stage training process where pre-training on unlabeled data continuously improves model capabilities, and then supervised fine-tuning continuously refines performance on specific tasks. This continuous learning approach maximizes the utility of all available data while reducing overall training time compared to starting from scratch with labeled data only.

The patent applies universality by designing a two-stage training framework where the first stage pre-trains a general-purpose model on unlabeled data from diverse sources, creating a versatile base model. The second stage then adapts this universal model to specific tasks through supervised fine-tuning. This allows the same pre-trained model to be reused across multiple different tasks, improving both adaptability and reducing redundant training.

The patent uses segmentation by separating the training process into two distinct phases: pre-training on unlabeled data and supervised fine-tuning on labeled data. This segmentation allows the model to first learn general, task-agnostic features that can be reused across multiple tasks, and then specialize for specific tasks only when needed, thereby improving model reusability while maintaining task performance.

The patent applies the taking out principle by extracting and utilizing the value in unlabeled data through pre-training, rather than discarding it or requiring full manual annotation. The system extracts general patterns and features from abundant unlabeled data, which would otherwise be wasted, and uses this extracted knowledge to reduce the amount of expensive labeled data needed for supervised fine-tuning, thereby reducing both cost and resource requirements.

The patent implements partial action by applying manual annotation only to a small subset of data needed for supervised fine-tuning, rather than annotating all training data. The majority of unlabeled data is used in the pre-training stage where no annotation is required, achieving high data quality for critical tasks while minimizing the quantity of manually annotated resources required.