Word Sequence Selection for Text Without Boundaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face excessive computational load when selecting an optimal word sequence in languages without word boundary markers, such as Chinese, due to the large variety of possible word sequences in long sentences.
Innovation Solution
A computer-implemented method and system that segments text to identify different word sequences, determines common word boundaries, and performs optimal word sequence selection for portions before these boundaries, reducing computational load by merging word sequences based on shared boundaries and avoiding incorrect segmentations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all possible word sequences are exhaustively listed and probabilities are computed for each possibility, then the accuracy of word sequence selection is improved, but the computational load becomes excessively large
Solution Approach 1:
The patent segments the text processing task by identifying common word boundaries that divide the text into independent units. By detecting positions where multiple word sequences converge to the same boundary point, the system divides the exhaustive search space into smaller, manageable segments that can be processed independently, thereby reducing computational load while maintaining accuracy.
Solution Approach 2:
The patent extracts and removes redundant computational paths by identifying common word boundaries. Once a common boundary is detected, the system extracts the shared segment and eliminates the need to recompute probabilities for identical sequences, taking out unnecessary computational overhead while preserving the accuracy of word sequence selection.
2Ease of operation
If traditional word segmentation methods are used without considering common boundaries, then the process is simpler, but the computational load increases excessively for long sentences
Solution Approach 1:
The patent performs preliminary action by pre-identifying common word boundaries before conducting exhaustive probability computations. By detecting convergence points where different segmentation paths lead to the same boundary, the system prepares the processing structure in advance, making subsequent computations more efficient and manageable without sacrificing simplicity.
Solution Approach 2:
The patent merges redundant computational paths by combining segments that share common word boundaries. When multiple word sequences converge to the same boundary position, the system merges their processing into a single operation, reducing the overall computational load while maintaining the simplicity of the segmentation approach.
3Ease of operation
If word sequences are processed independently without merging at common boundaries, then the processing is more straightforward, but the efficiency of text processing applications is reduced
Solution Approach 1:
The patent merges word sequence processing at common boundaries by detecting positions where different segmentation paths converge. When a common boundary is identified, the system combines the processing of multiple sequences into a unified operation, maintaining straightforward processing logic while significantly improving the efficiency of text processing applications such as search engines and machine translation.
Data Source
AI summary
The present disclosure discloses a method and apparatus of selecting a word sequence for a text written in a language without word boundary in order to solve the problem of having excessively large computation load when selecting an optimal word sequence in existing technologies. The disclosed method includes: segmenting a segment of the text to obtain different word sequences; determining a common word boundary for the word sequences; and performing optimal word sequence selection for portions of the word sequences prior to the common word boundary. Because optimal word sequence selection is performed for portions of word sequences prior to a common word boundary, shorter independent units can be obtained, thus reducing computation load of word segmentation.