Word Segmentation Using Scaled Probabilities
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional word segmentation techniques favor longer words over shorter words due to independent probability assumptions, leading to inaccurate segmentations.
Innovation Solution
The use of scaled probabilities is introduced, where a scaling weight is applied based on the number of atomic units in an n-gram, allowing for more accurate segmentation by adjusting the probability of n-grams to prevent favoring longer words over shorter ones.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional independent probability assumption is used for word segmentation, then the segmentation process is simple and fast, but the segmentation accuracy deteriorates because longer words are unfairly favored over shorter words
Solution Approach 1:
The patent changes the probability parameter by introducing scaled probabilities that adjust the weight of each n-gram based on its length. Instead of using raw independent probabilities, the system applies scaling factors that compensate for the positional bias, transforming the probability calculation to achieve more accurate segmentation results while maintaining computational feasibility.
Solution Approach 2:
The patent performs preliminary scaling of probabilities before the actual segmentation decision is made. By pre-calculating and storing scaled probabilities for all n-grams in the dictionary, the system prepares adjusted probability values in advance, allowing the segmentation algorithm to use these corrected values without adding significant computational complexity during runtime.
2Measurement precision
If scaled probabilities are applied to correct the bias toward longer words, then segmentation accuracy improves, but computational complexity increases
Solution Approach 1:
The system performs the computationally intensive probability scaling operation in advance during dictionary construction. All scaled probabilities are pre-calculated and stored in the segmentation dictionary, so that during actual text segmentation, the system only needs to retrieve and compare pre-computed values, avoiding repeated complex calculations and maintaining fast processing speed.
3Ease of manufacture
If traditional probability multiplication method is used, then the calculation is straightforward, but the result is biased toward segmentations with more atomic units
Solution Approach 1:
The patent modifies the probability parameter by introducing scaling factors that adjust for n-gram length. The scaled probability is computed as the product of the original probability and a scaling factor that depends on the n-gram length, thereby correcting the bias toward longer words while maintaining the multiplicative calculation structure for ease of implementation.
Data Source
AI summary
Systems, methods, and apparatuses including computer program products for segmenting words using scaled probabilities. In one implementation, a method is provided. The method includes receiving a probability of a n-gram identifying a word, determining a number of atomic units in the corresponding n-gram, identifying a scaling weight depending on the number of atomic units in the n-gram, and applying the scaling weight to the probability of the n-gram identifying a word to determine a scaled probability of the n-gram identifying a word.