Columnar Data Fingerprints for Block Pruning in Compressed Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High data compression ratios in in-memory databases lead to memory and performance overhead during data queries, as decompressing entire datasets is resource-intensive and consumes extra CPU and memory resources.
Innovation Solution
Implementing fine-granularity block-level pruning using Fingerprint Headers that maintain approximate data ranges and gaps, allowing for efficient block pruning and reducing the need for decompressing all data blocks during queries by using orthogonal fingerprints independent of compression techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If high data compression ratio is used to store large amount of data in-memory, then memory usage is reduced, but CPU and memory overhead for data queries increases due to decompression requirements
Solution Approach 1:
The patent segments the columnar data into multiple data blocks, each with its own fingerprint header. This segmentation allows the system to selectively decompress only the necessary blocks during queries rather than decompressing the entire dataset, thereby reducing CPU and memory overhead while maintaining the memory space savings of compression.
Solution Approach 2:
The patent performs preliminary action by generating fingerprint headers for each data block during the data loading phase. These fingerprints capture approximate data ranges and gaps in advance, enabling efficient query filtering without decompression. This preliminary processing reduces the computational overhead during actual query operations.
2Reliability
If entire compressed data is decompressed for each query, then query accuracy is improved, but decompression overhead and resource consumption increase
Solution Approach 1:
The patent extracts essential information from each data block by generating fingerprint headers that capture approximate data ranges and gaps. These extracted fingerprints are stored with the compressed data and used to filter out irrelevant blocks before decompression, ensuring that only potentially relevant blocks are decompressed for query processing.
Solution Approach 2:
The patent introduces fingerprint headers as an intermediary between the compressed data and the query processing system. These fingerprints act as a filtering layer that enables accurate block selection without full decompression, reducing decompression overhead while maintaining query accuracy through precise range and gap matching.
3Productivity
If block pruning is implemented using fingerprints, then false positive rates are reduced and decompression overhead is minimized, but additional memory overhead for storing fingerprints is introduced
Solution Approach 1:
The patent uses parameter changes by representing data block characteristics through compact fingerprint parameters that capture approximate ranges and gaps. These parameterized representations enable efficient block pruning with minimal memory overhead, as the fingerprints are significantly smaller than the actual compressed data blocks they describe.
Data Source
AI summary
The present disclosure involves systems, software, and computer implemented methods for compressed columnar data search using fingerprints. One example method includes compressing columnar data that includes dividing the columnar data into multiple data blocks and generating a fingerprint for each data block, storing the compressed columnar data and the generated fingerprints in an in-memory database, receiving a query for the columnar data, for each in-memory data block stored in the in-memory database, determining whether the in-memory data block satisfies the query and in response to a determination that the in-memory data block does not satisfy the query, pruning the in-memory data block from the multiple data blocks to generate an unpruned set of data blocks, decompressing the unpruned set of data blocks, and performing a query search on the decompressed unpruned set of data blocks for the received query.