Text Sequence Segmentation via Multi-Model Probability Combination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Chinese text sequence segmentation models face challenges in accurately segmenting new texts due to insufficient and outdated corpora, leading to low acquisition efficiency and high costs, with easy data augmentation methods failing to address out-of-vocabulary issues effectively.
Innovation Solution
A method involving multiple segmentation models to segment a text sequence, combining state values to determine segmentation probabilities, and using a result combination model with weights to improve segmentation accuracy, allowing for automatic annotation and corpus expansion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple segmentation models are used to segment text sequences, then segmentation accuracy is improved, but device complexity increases
Solution Approach 1:
The patent divides the text segmentation task into multiple independent segmentation models, each processing the input text sequence separately to generate segmentation sub-results. This segmentation of the overall task into parallel sub-tasks allows for improved accuracy through result combination while maintaining manageable complexity of individual models
Solution Approach 2:
The patent combines the segmentation results from multiple segmentation models through a result combination module. By merging the outputs of multiple models and determining final segmentation positions based on aggregated results, the system achieves higher segmentation accuracy than any single model could provide alone
2Productivity
If easy data augmentation methods are used, then productivity is improved, but measurement precision deteriorates due to out-of-vocabulary issues
Solution Approach 1:
The patent enables the segmentation system to automatically annotate new text sequences and expand the training corpus without manual intervention. By using the multiple segmentation models to process new texts and generate annotated data automatically, the system serves itself to continuously improve its training data while maintaining high productivity and avoiding out-of-vocabulary problems through diverse model outputs
Data Source
AI summary
The present disclosure, belonging to the technical field of natural language processing, provides a text sequence segmentation method. The method can include acquiring n segmentation sub-results of the text sequence, the n segmentation sub-results being acquired by segmenting the text sequence by n segmentation models, and processing the n segmentation sub-results by a probability determination model branch in a result combination model to acquire a segmentation probability of the each segmentation position. The method can further include processing the segmentation probability of the each segmentation position by an activation function in the result combination model to acquire a segmentation result of the text sequence. According to the present disclosure, using each segmentation position in the text sequence as a unit, segmentation results of a plurality of segmentation models are combined, such that the accuracy of segmentation of a new text can be improved.