Word Vector Processing for Chinese Using N-gram Stroke Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional word vector generation algorithms, primarily designed for English and Romance languages, produce poor results when applied to Chinese, failing to accurately represent semantic information of Chinese words.
Innovation Solution
The proposed solution involves using n-gram strokes to represent Chinese words, where each n-gram stroke represents successive strokes of a character, and employing a computer-implemented method to initialize and train word vectors and stroke vectors using a neural network, ensuring improved accuracy of word vectors by leveraging the unique structure of Chinese characters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional word vector generation algorithms (designed for English/Romance languages) are applied to Chinese, then the device complexity remains simple, but the measurement precision of word vectors deteriorates
Solution Approach 1:
The patent segments Chinese characters into n-gram strokes (e.g., 1-gram, 2-gram, 3-gram strokes) to represent character structures. This segmentation allows the algorithm to capture structural features of Chinese characters systematically, improving word vector accuracy without requiring overly complex processing mechanisms.
Solution Approach 2:
The patent introduces stroke-based parameters (n-gram stroke sequences) as new features for representing Chinese characters. By changing the representation parameters from whole-character encoding to stroke-level encoding, the algorithm achieves better measurement precision while maintaining manageable complexity through systematic parameter organization.
2Measurement precision
If n-gram strokes are used to represent Chinese words, then the measurement precision of word vectors improves, but the device complexity increases
Solution Approach 1:
By segmenting Chinese characters into standardized stroke sequences (n-grams), the patent transforms complex character recognition into systematic pattern matching. The segmentation principle organizes the complexity into manageable units that can be processed efficiently through neural network layers.
Solution Approach 2:
The patent transitions from traditional character-level representation to stroke-level representation, adding a new dimensional perspective. This dimensionality change allows the system to capture structural relationships that were invisible at the character level, improving accuracy while the stroke sequences provide a regularized structure that manages computational complexity.
3Reliability
If conventional algorithms are used for Chinese language processing, then the ease of operation is maintained, but the reliability of word vector representation deteriorates
Solution Approach 1:
The patent creates a universal stroke-based representation system that can handle different Chinese characters consistently. The n-gram stroke approach provides a multi-functional framework that works across various Chinese characters and contexts, improving reliability while the underlying neural network architecture maintains ease of operation through standardized processing pipelines.
Solution Approach 2:
By changing the fundamental representation parameters to stroke sequences, the patent achieves more reliable word vector representations that capture semantic and structural information. The parameter change is implemented through systematic preprocessing steps that maintain operational simplicity while dramatically improving representation reliability.
Data Source
AI summary
A word vector processing method is provided. Word segmentation is performed on a corpus to obtain words, and n-gram strokes corresponding to the words are determined. Each n-gram stroke represents n successive strokes of a corresponding word. Word vectors of the words and stroke vectors of the n-gram strokes are initialized corresponding to the words. After performing the word segmentation, the n-gram strokes are determined, and the word vectors and stroke vectors are determined, training the word vectors and the stroke vectors.