Multi-Granularity Text Segmentation via Lexicon Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing word segmentation technologies face challenges in providing multiple granularity levels effectively, leading to low recall rates and semantic item loss, as well as the need for vast and costly lexicon maintenance due to ambiguity and precision issues.
Innovation Solution
A system that establishes a lexicon of smallest semantic units with reasonable lengths and semantic integrity, allowing for medium-grained segmentation results to be merged into coarser-grained results, and using search elements to obtain finer-grained results, thereby addressing the ambiguity and precision issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If maximum matching method is used for word segmentation, then segmentation speed is improved, but segmentation precision deteriorates due to ambiguity
Solution Approach 1:
The patent segments the word segmentation process into two distinct stages: first performing maximum matching segmentation to obtain initial segments, then performing ambiguity resolution segmentation to refine the results. This two-stage segmentation approach allows the system to maintain high processing speed while improving segmentation precision by addressing ambiguities in a separate refinement step.
Solution Approach 2:
The patent performs maximum matching segmentation as a preliminary action before ambiguity resolution. By first obtaining initial segmentation results through fast maximum matching, the system prepares a baseline that can then be refined. This preliminary segmentation provides a starting point that speeds up the overall process while allowing subsequent refinement to improve precision.
2Reliability
If finer-grained segmentation is used, then search recall rate is improved, but search precision deteriorates
Solution Approach 1:
The patent implements dynamic granularity adjustment where the segmentation granularity is not fixed but adapts based on the specific text and segmentation context. The system can dynamically switch between finer-grained and coarser-grained segmentation strategies, allowing it to optimize the balance between recall rate and precision for different search scenarios and text types.
Solution Approach 2:
The patent changes the granularity parameter of segmentation based on different search requirements and text characteristics. By adjusting the granularity parameter dynamically, the system can optimize search recall rate when finer-grained segmentation is needed while maintaining search precision when coarser-grained segmentation is more appropriate, thus resolving the contradiction between these two metrics.
3Measurement precision
If vast lexicon is used to improve segmentation precision, then segmentation precision is improved, but lexicon maintenance complexity and cost increase
Solution Approach 1:
The patent segments the lexicon into different levels and types, organizing lexical resources in a hierarchical structure. This segmentation of the lexicon allows the system to use comprehensive lexical coverage for precision without requiring a single vast flat lexicon, thereby reducing maintenance complexity through structured organization.
Solution Approach 2:
The patent introduces an intermediary ambiguity resolution mechanism that acts as a mediator between the segmentation process and the lexicon. This intermediary layer handles complex disambiguation tasks, allowing the system to achieve high segmentation precision without relying solely on an excessively large lexicon, thus reducing lexicon maintenance burden while maintaining precision.
4Measurement precision
If coarser-grained segmentation is used, then search precision is improved, but search recall rate deteriorates
Solution Approach 1:
The patent implements dynamic granularity adjustment that allows the system to adapt between coarser-grained and finer-grained segmentation based on search requirements. When search precision is prioritized, coarser-grained segmentation is used; when search recall rate is prioritized, finer-grained segmentation is applied. This dynamic approach resolves the contradiction by making granularity adjustable rather than fixed.
Solution Approach 2:
The patent creates a universal segmentation system that can serve multiple search objectives simultaneously. By implementing multi-granularity segmentation capabilities, the system can provide both coarser-grained results for precision-oriented searches and finer-grained results for recall-oriented searches, making the segmentation system universally applicable to different search scenarios without sacrificing either precision or recall rate.
Data Source
AI summary
Text processing includes: segmenting received text based on a lexicon of smallest semantic units to obtain medium-grained segmentation results; merging the medium-grained segmentation results to obtain coarse-grained segmentation results, the coarse-grained segmentation results having coarser granularity than the medium-grained segmentation results; looking up in the lexicon of smallest semantic units respective search elements that correspond to segments in the medium-grained segmentation results; and forming fine-grained segmentation results based on the respective search elements, the fine-grained segmentation results having finer granularity than the medium-grained segmentation results.