Metadata Search via Partial N-Gram Indexing for Large Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing searches in very large tables is challenging due to the significant time and computing resources required to scan the entire table, leading to slow and resource-intensive search operations.
Innovation Solution
Implementing a N-Gram index that allows for faster search results by creating an index that includes partial N-Gram components and utilizing recent log data, along with techniques for building and efficiently searching an index store.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a full table scan is performed to ensure complete search coverage, then search accuracy is improved, but search speed and resource efficiency deteriorate
Solution Approach 1:
The patent segments the large table into multiple partitions or blocks, and creates separate indexes for each partition. This allows the search to only scan relevant partitions rather than the entire table, maintaining search accuracy while significantly improving search speed by reducing the scanned data scope.
Solution Approach 2:
The patent introduces an index structure as an intermediary between the search query and the full table data. The index contains pre-computed N-gram signatures that enable rapid filtering of candidate records before performing full text matching, thus achieving both high search accuracy and fast search performance.
2Productivity
If traditional indexing methods are used to improve search speed, then search efficiency is improved, but memory usage and index building complexity worsen
Solution Approach 1:
The patent transforms the full text data into N-gram signatures with a specific parameter (N-gram size and step), creating a compressed representation that reduces index size and simplifies the indexing process. This parameter-based transformation enables efficient search while reducing the complexity of index construction and management.
3Loss of time
If N-gram indexing is implemented to reduce search time, then search speed is improved, but index construction time and computational resources worsen
Solution Approach 1:
The patent performs preliminary N-gram signature computation and stores these signatures in the index structure during data ingestion or periodic updates. This preliminary action ensures that when search queries arrive, the N-gram matching can be performed rapidly without requiring real-time computation, thus reducing search time while distributing the computational burden over time.
Data Source
AI summary
As described herein, a N-Gram index may be created and the search may be conducted using the index, which will lead to faster search results. The N-Gram index may also include partial N-Gram components to capture more relevant data. Moreover, as described herein, the search may also take into account recent log data that has not yet been indexed. Techniques for building an index store using log data and efficiently searching the index store and log data to process search requests are described herein.