Neural Character Language Model for Word Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Supervised word segmentation approaches fail to accurately segment product titles in languages like Asian languages, leading to incorrect categorization and hindering online shopping experiences by not properly obeying natural language grammars.
Innovation Solution
A computer-implemented system using a competing neural character language model for unsupervised word segmentation, which converts characters into embedding vectors and feeds them into forward and reverse language models to determine accurate word boundaries based on Euclidean distance comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If supervised word segmentation approaches are used, then product titles can be segmented into keywords, but the segmentation accuracy deteriorates for Asian languages and fails to obey natural language grammars
Solution Approach 1:
The system performs unsupervised word segmentation by automatically learning from raw product title text without requiring manually annotated training data. The neural character language model self-adjusts its parameters during training to maximize likelihood of generating the observed text sequences, thereby achieving accurate word segmentation for Asian languages without supervised guidance
Solution Approach 2:
The patent transforms the word segmentation problem into a language modeling task by changing the optimization parameter from classification accuracy to likelihood maximization. The model learns character-level transitions and uses the log-likelihood ratio to identify word boundaries, adapting to different Asian languages through this parameter transformation rather than language-specific rules
2Ease of operation
If supervised word segmentation approaches are used, then segmentation can be performed, but the system complexity increases due to requirement of annotated training data
Solution Approach 1:
The system eliminates the need for manually annotated training data by performing unsupervised learning directly on raw product titles. The neural model automatically discovers word boundaries and language patterns through self-supervised training, reducing system complexity by removing the data annotation infrastructure
Solution Approach 2:
The patent performs character embedding preprocessing to convert input characters into dense vector representations before feeding them to the language model. This preliminary transformation simplifies the subsequent segmentation task by capturing character-level patterns that facilitate automatic word boundary detection
3Productivity
If improper word segmentation is performed, then product categorization can be completed, but the recommendation quality and search accuracy deteriorate
Solution Approach 1:
The system uses the log-likelihood ratio as a feedback signal to identify word boundaries. By comparing the likelihood of generating a character given previous characters versus the reverse direction, the model receives feedback that guides it to correctly segment words while maintaining high categorization productivity
Solution Approach 2:
The patent replaces traditional mechanical segmentation rules with a neural character language model that learns contextual patterns. This substitution enables the system to maintain high processing speed while improving search accuracy through learned linguistic patterns rather than rigid rule-based approaches
Data Source
AI summary
Systems and methods are provided for detecting inaccuracy in a product title, comprising identifying, by running a string algorithm on a title associated with a product, at least one product type associated with the product, predicting, using a machine learning algorithm, at least one product type associated with the product based on the title, detecting an inaccuracy in the title, based on at least one of the identification or the prediction, and outputting, to a remote device, a message indicating that the title comprises the inaccuracy. Running the string algorithm may comprise receiving a set of strings, generating a tree based on the received set of strings, receiving the title, and traversing the generated tree using the title to find a match. Using the machine learning algorithm may comprise identifying words in the title, learning a vector representation for each character n-gram of each word, and summing each character n-gram.