Query Segmentation Using Word Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current query segmentation techniques in search engines are not accurate enough to identify meaningful phrases in search queries, leading to suboptimal search results, as they fail to effectively determine the proper associations between tokens in queries.
Innovation Solution
The use of word embeddings models like word2vec and GloVe to generate vector representations of tokens, which are then processed by a binary classifier to determine whether tokens should be included in the same query segment, based on their context and frequency of co-occurrence in training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional query segmentation techniques are used, then the system is simpler to implement, but the segmentation accuracy is insufficient
Solution Approach 1:
The patent replaces traditional mechanical/token-based segmentation methods with a neural network-based system that uses word embeddings and deep learning models to achieve more accurate query segmentation by learning semantic relationships between words
Solution Approach 2:
The patent transforms the segmentation approach by changing from discrete token-based processing to continuous vector space representations, using parameters like word embeddings, attention mechanisms, and neural network layers to improve segmentation accuracy
2Reliability
If word embeddings models are used to improve segmentation accuracy, then query processing becomes more complex, but search result relevance improves
Solution Approach 1:
The patent performs preliminary actions by pre-training word embedding models and segmentation networks during an offline phase, so that during online query processing, the system only needs to apply the pre-learned models rather than training them in real-time, reducing operational complexity
Solution Approach 2:
The patent introduces word embedding vectors as an intermediary representation between raw query tokens and the segmentation decision, allowing the system to capture semantic meaning and contextual relationships without directly processing complex linguistic rules
Data Source
AI summary
A query segmentation system segments a search query into a query segment that identifies search results from items stored on a database. The query segmentation system receives a query string, and the query string comprises a plurality of tokens. A first token and a second token are identified from the plurality of tokens, and a first vector and a second vector associated with the first token and the second token respectively are determined. The query segmentation system determines whether to include the first and second tokens in a single query segment based on the first and second vectors. The single query segment is processed to identify the search results from the database.