Customizing Word Segmentation Model via Increment Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Chinese word segmentation models are inefficient and labor-intensive to customize for professional fields or specific scenes, often requiring re-training and struggling to achieve accurate segmentation.
Innovation Solution
A method utilizing a customized segmentation training corpus to rectify a preset word segmentation model through increment training or weight intervention, generating a tailored model that meets specific requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a common word segmentation model is re-trained with a segmentation training corpus of the professional field or specific scene, then the word segmentation model can be customized for professional fields, but it takes much time and labor
Solution Approach 1:
The patent pre-trains a word segmentation model on a large-scale general corpus before fine-tuning. This preliminary training establishes a robust base model that can be quickly adapted to specific domains through subsequent fine-tuning with domain-specific corpora, significantly reducing the time required for customization compared to training from scratch.
Solution Approach 2:
The training process is divided into two distinct stages: pre-training on general corpora and fine-tuning on domain-specific corpora. This segmentation of the training process allows the model to first learn general language patterns efficiently, then specialize for specific domains with minimal additional time investment.
2Adaptability or versatility
If a common word segmentation model is re-trained with a segmentation training corpus of the professional field or specific scene, then the word segmentation model can be customized for professional fields, but it is difficult to guarantee segmentation quality
Solution Approach 1:
The model undergoes preliminary training on extensive general corpora to establish a strong foundation of language understanding and word boundary detection capabilities. This pre-training ensures that the model starts with high baseline performance before domain-specific fine-tuning, making it easier to maintain high segmentation quality across different domains.
Solution Approach 2:
The patent employs fine-tuning that adjusts model parameters based on domain-specific characteristics while preserving the general language understanding learned during pre-training. This controlled parameter adjustment allows the model to adapt to professional fields while maintaining overall segmentation quality through the transfer of general linguistic knowledge.
3Ease of operation
If a common word segmentation model is used, then the model can handle general text, but it cannot accurately perform the word segmentation in a professional field or specific scene
Solution Approach 1:
The patent creates a multi-functional word segmentation model through pre-training on diverse general corpora that cover various domains and text types. This universal base model can handle general text effectively and can be subsequently adapted to specific professional fields through fine-tuning, thus achieving both general applicability and domain-specific accuracy.
Solution Approach 2:
The model uses parameter adjustment during fine-tuning to specialize for specific domains while maintaining the general capabilities learned during pre-training. This allows the same model architecture to serve multiple functions - handling both general and professional text with appropriate accuracy by changing only the trained parameters rather than the overall model structure.
Data Source
AI summary
Embodiments of the present disclosure disclose a method and an apparatus for customizing a word segmentation model based on artificial intelligence, a device and a medium. The method includes the followings. A customized segmentation training corpus is acquired. A first preset word segmentation model is rectified with an increment training method or a weight intervention method, based on the customized segmentation training corpus, to obtain a customized word segmentation model corresponding to the customized segmentation training corpus.