N-Gram Index Search for Large-Table Query Speed and Coverage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing searches in very large tables is challenging due to the significant time and computing resources required to scan the entire table, leading to slow and resource-intensive search operations.
Innovation Solution
Implementing a N-Gram index that allows for faster search results by creating an index that includes partial N-Gram components and utilizing recent log data, along with techniques for building and efficiently searching an index store.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a full table scan is performed to ensure complete search coverage, then search accuracy is improved, but search speed and resource efficiency deteriorate
Solution Approach 1:
The patent segments the search process into two distinct phases: (1) rapid candidate identification using an N-gram index that stores pre-computed n-gram signatures of document blocks, and (2) precise verification by retrieving and scanning only the candidate blocks identified in phase 1. This segmentation allows the system to achieve both fast initial filtering and accurate final verification, resolving the contradiction between search speed and accuracy.
2Reliability
If the entire table is scanned to ensure complete data coverage, then search completeness is improved, but computing resource consumption increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing N-gram signatures for all document blocks in the database during an offline indexing phase. This preliminary processing creates a compact index structure that enables rapid candidate identification during online search operations, eliminating the need to scan entire tables during query execution while maintaining complete search coverage.
3Device complexity
If traditional search methods are used to maintain simplicity, then system complexity is reduced, but search efficiency deteriorates
Solution Approach 1:
The patent introduces an N-gram index as an intermediary data structure between the full text database and the search query processor. This index acts as a mediator that translates arbitrary search queries into efficient candidate block identification, enabling fast search operations without requiring fundamental changes to the underlying database architecture or query processing framework.
Data Source
AI summary
As described herein, a N-Gram index may be created and the search may be conducted using the index, which will lead to faster search results. The N-Gram index may also include partial N-Gram components to capture more relevant data. Moreover, as described herein, the search may also take into account recent log data that has not yet been indexed. Techniques for building an index store using log data and efficiently searching the index store and log data to process search requests are described herein.