Search Query Chunking With Dictionary Matching and CharacterBERT
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing chunking technologies for search queries, such as morphological and syntactic analysis, fail to achieve sufficient accuracy when applied to specific search queries due to assumptions based on general words, leading to poor chunking of search queries like 'The South Mountain' or 'Tokyorestaurant', where spaces are incorrectly input or missing.
Innovation Solution
A chunking execution system that combines dictionary data, a collocation score, and a learning model, specifically using CharacterBERT, to accurately determine tokenization and chunking of search queries by analyzing token attributes and relevance, adjusting tokens through division or combination based on dictionary matches and learning model outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If general morphological analysis or syntactic analysis is used for chunking search queries, then the processing can be performed with existing technologies, but the chunking accuracy is insufficient for specific search queries
Solution Approach 1:
The patent applies local quality by training the learning model specifically on search query data rather than using general-purpose NLP models. The model learns patterns specific to search queries, such as brand names, product terms, and user search behaviors, enabling accurate chunking for this specific domain while maintaining the ability to handle various search query types
Solution Approach 2:
The patent changes the fundamental approach from rule-based morphological/syntactic analysis to data-driven learning models. By training on labeled search query data, the system adapts its parameters to capture the specific patterns and structures of search queries, achieving superior accuracy compared to general-purpose methods
2Measurement precision
If machine learning models for named entity recognition are applied to search query chunking, then extraction accuracy for named entities can be improved, but the model is not designed for chunking and assumes sentences of certain length
Solution Approach 1:
Instead of applying general NER models to search queries and hoping they work, the patent inverts the approach by training a learning model specifically on search query chunking tasks. The model is trained on labeled data where the correct chunking of search queries is known, allowing it to learn the specific patterns of search queries rather than forcing search queries into a general NER framework
Solution Approach 2:
The patent makes the system dynamic by using a learning model that can adapt to different search query structures and lengths. Unlike fixed-length assumptions in traditional NER, the learning model processes search queries of varying lengths and structures flexibly, adjusting its predictions based on the specific input while maintaining high accuracy
3Productivity
If search queries are processed without proper chunking, then processing can be simpler and faster, but search results precision deteriorates due to incorrect token formation
Solution Approach 1:
The patent applies preliminary action by performing accurate chunking before the actual search processing. The learning model pre-processes the search query to correctly identify and separate tokens, ensuring that subsequent search operations work with properly formatted queries. This upfront chunking prevents errors later in the processing pipeline while maintaining efficiency
Data Source
AI summary
A chunking execution system including at least one processor which acquires a search query including at least one token; executes matching with use of dictionary data based on the search query; acquires an output from a learning model on chunking based on the search query; and executes chunking on the search query based on a result of executing the matching and the output from the learning model.