Columnar Data Fingerprints for Block Pruning in Compressed Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High data compression ratios in in-memory databases lead to memory and performance overhead during data queries, as decompressing entire datasets is resource-intensive and results in high false positive rates during searches.
Innovation Solution
Implementing fine-granularity column block-level pruning using Fingerprint Headers that maintain approximate data ranges and gaps, allowing for efficient block pruning and reducing the need for full decompression by using orthogonal fingerprints independent of compression techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If high data compression ratio is used to store large amount of data in-memory, then memory efficiency is improved, but query performance deteriorates due to decompression overhead
Solution Approach 1:
The patent divides compressed columnar data into multiple data blocks, each with its own fingerprint header. This segmentation allows the system to selectively decompress only relevant blocks during queries rather than the entire dataset, thus maintaining high compression ratios while improving query performance through partial decompression.
Solution Approach 2:
The patent pre-computes and stores fingerprint headers for each data block during the data loading phase. These fingerprints contain summary information about the block's data range and distribution. During queries, these pre-computed fingerprints enable rapid determination of which blocks need decompression, eliminating the need for full decompression and reducing query overhead.
2Reliability
If full decompression is performed for data queries, then query accuracy is improved, but CPU and memory resource consumption increases
Solution Approach 1:
The patent extracts essential summary information from each data block to create compact fingerprint headers. These fingerprints capture the minimum and maximum values, data distribution characteristics, and other metadata needed to determine query relevance. By extracting only this essential information, the system avoids full decompression while maintaining query accuracy through fingerprint-based filtering.
Solution Approach 2:
The patent applies different processing quality to different data blocks based on their relevance to the query. Blocks that pass the fingerprint filter undergo full decompression for accurate query processing, while blocks that fail the filter are pruned without decompression. This local quality approach ensures query accuracy for relevant data while minimizing resource consumption for irrelevant data.
3Measurement precision
If data is divided into multiple blocks with fingerprints for pruning, then false positive rates are reduced, but device complexity increases
Solution Approach 1:
The patent creates lightweight fingerprint copies of each data block's essential characteristics. These fingerprints are compact metadata structures that replicate the minimum and maximum values, data type information, and distribution patterns of the original blocks. This copying approach enables accurate block pruning without the complexity of analyzing entire data blocks, reducing false positives while maintaining manageable system complexity.
Data Source
AI summary
The present disclosure involves systems, software, and computer implemented methods for compressed columnar data search using fingerprints. One example method includes compressing columnar data that includes dividing the columnar data into multiple data blocks and generating a fingerprint for each data block, storing the compressed columnar data and the generated fingerprints in an in-memory database, receiving a query for the columnar data, for each in-memory data block stored in the in-memory database, determining whether the in-memory data block satisfies the query and in response to a determination that the in-memory data block does not satisfy the query, pruning the in-memory data block from the multiple data blocks to generate an unpruned set of data blocks, decompressing the unpruned set of data blocks, and performing a query search on the decompressed unpruned set of data blocks for the received query.