Query Processing Using Variable-Length N-gram Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text document search methods are inefficient in processing query strings, particularly when dealing with sequences of varying lengths, leading to increased query processing time and reduced search performance.
Innovation Solution
The method generates an effective string set from a query string by separating it into n-grams of varying lengths, selecting unique n-grams from an index term database, and determining a candidate subset based on access costs to minimize search costs, using a tree structure and depth-first search to optimize the search process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a query string is processed using traditional search methods, then the search can be performed, but the query processing time increases and search performance decreases
Solution Approach 1:
The query string is segmented into multiple partial strings of different lengths (n-grams). This segmentation allows the system to process the query by searching for these smaller units independently, which reduces the overall query processing time and improves search performance while maintaining completeness.
Solution Approach 2:
The system performs partial actions by searching for partial strings rather than requiring exact matches of the entire query string. This partial action approach allows for faster processing and can retrieve relevant results more efficiently, addressing the time loss issue without sacrificing search quality.
2Productivity
If n-grams of varying lengths are used, then search performance improves, but the complexity of processing increases
Solution Approach 1:
By segmenting the query into n-grams of varying lengths, the system can leverage existing index structures more effectively. This segmentation strategy improves search performance by allowing parallel processing and reducing the computational burden on any single operation, thereby managing complexity while enhancing productivity.
Solution Approach 2:
The system changes the parameter of string length by using n-grams with different lengths rather than fixed-length tokens. This parameter change allows the system to adapt to different query patterns and improve search performance without requiring a complete redesign of the processing architecture, thus managing complexity effectively.
3Reliability
If all partial strings are processed, then search completeness is maintained, but processing time increases
Solution Approach 1:
The system uses partial action by processing only the necessary partial strings (n-grams) rather than all possible substrings. This selective processing maintains search completeness for relevant queries while significantly reducing processing time, resolving the contradiction between reliability and time loss.
Solution Approach 2:
The system performs preliminary action by pre-processing the query string into n-grams of varying lengths before executing the search. This preliminary segmentation allows the search to proceed more efficiently by having the data ready in an optimized format, thereby reducing processing time while maintaining search completeness.
Data Source
AI summary
Provided are a method and apparatus for processing a query. The method includes generating string sets comprising a plurality of partial strings from a query string, determining a subset of the string sets as a candidate set, and searching for a document comprising the query string from the candidate set.