Word Segmentation Confidence Threshold Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current word segmentation systems are inflexible and resource-intensive, as they are designed for specific applications and struggle to adapt to different scenarios, leading to resource waste and maintenance difficulties, with varying requirements for precision and speed across applications like information retrieval and machine translation.
Innovation Solution
A word segmentation method and system that adjusts precision by using a confidence level threshold, employing a linear classifier for rapid classification and combining results from simple and complex segmentation modules, with features like word length, dictionary type, and context, to adapt to multiple application scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If different word segmentation systems are customized for different applications, then application-specific requirements are met, but resource waste and system maintenance difficulty increase
Solution Approach 1:
The patent creates a universal word segmentation system that can serve multiple applications by dynamically adjusting segmentation strategies based on application requirements. The system uses a configuration file to store different application scenarios and their corresponding parameters, allowing one system to fulfill multiple functions rather than requiring separate customized systems for each application.
Solution Approach 2:
The system dynamically adjusts word segmentation behavior by changing confidence level thresholds and selection criteria based on the active application scenario. This dynamic adaptation allows the same system to optimize for speed in information retrieval applications while prioritizing accuracy in machine translation applications, without requiring separate static systems for each use case.
2Measurement precision
If word segmentation precision is increased for machine translation, then translation accuracy improves, but word segmentation speed and consistency decrease
Solution Approach 1:
The system changes the confidence level threshold parameter based on the application scenario. For machine translation, a higher confidence level threshold is set to ensure accuracy, while for information retrieval, a lower threshold allows faster processing. This parameter adjustment resolves the contradiction by allowing the system to prioritize precision or speed depending on the active scenario.
Solution Approach 2:
The system applies different quality standards to different application contexts. Instead of using a single fixed precision level for all applications, the system tailors the segmentation quality to match local requirements - high precision for machine translation where accuracy is critical, and lower precision for information retrieval where speed is more important.
3Productivity
If word segmentation speed is increased for information retrieval, then processing efficiency improves, but word segmentation correctness and recognition rate decrease
Solution Approach 1:
The system adjusts the confidence level threshold parameter downward for information retrieval applications, allowing faster processing by accepting lower confidence decisions. This parameter change enables the system to prioritize speed over precision when the active scenario is information retrieval, resolving the contradiction between speed and correctness.
Solution Approach 2:
The system applies partial action by performing only the necessary level of analysis for each application type. For information retrieval, it uses a simplified, faster segmentation approach without the full rigor applied in machine translation, achieving sufficient correctness for the task while maximizing speed.
Data Source
Figure 1
Figure 2
Figure 3~5
AI summary
Embodiments of the present invention provide a word segmentation method and system for a language text. The method includes: obtaining a first language text to be processed and a confidence level threshold; performing word segmentation on the first language text in a first word segmentation manner, to obtain a first word boundary set; dividing, according to the confidence level threshold, the first word boundary set into a trusted second word boundary set and an untrusted third word boundary set; selecting a second language text from the first language text according to the third word boundary set, where the second language text includes a word corresponding to each word boundary in the third word boundary set; performing word segmentation on the second language text in a second word segmentation manner, to obtain a fourth word boundary set; and determining the second word boundary set and the fourth word boundary set as a word segmentation result of the first language text. Word segmentation precision required by the first language text can be flexibly adjusted by adjusting the confidence level threshold, so as to adapt to multiple application scenarios requiring different word segmentation precision.