Columnar Data Fingerprints for Selective Query Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High data compression ratios in in-memory databases lead to memory and performance overhead during data queries, as decompressing entire datasets is resource-intensive and consumes extra CPU and memory resources.
Innovation Solution
Implementing fine-granularity column block-level pruning using Fingerprint Headers that maintain approximate data ranges and gaps, allowing for efficient block pruning and reducing the need for decompressing all data blocks during queries, thereby minimizing decompression overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If high data compression ratio is used to store large amount of data in-memory, then memory efficiency is improved, but decompression overhead and CPU resource consumption increase during data queries
Solution Approach 1:
The patent divides columnar data into multiple data blocks and generates a fingerprint for each block. During query processing, the system segments the decompression operation by only decompressing blocks that may contain matching data, rather than decompressing the entire compressed dataset. This segmentation approach reduces CPU resource consumption during queries while maintaining high compression ratios for storage.
Solution Approach 2:
The patent performs preliminary action by generating fingerprints for each data block during the compression phase. These fingerprints are stored alongside the compressed data, enabling the system to quickly determine which blocks need decompression before the actual query executes. This preliminary preparation reduces the computational overhead during query processing.
2Reliability
If entire compressed data is decompressed for each query, then query accuracy is improved, but memory overhead and query performance deteriorate
Solution Approach 1:
The patent applies local quality by using fingerprints to identify specific data blocks that are relevant to the query. Instead of decompressing the entire dataset, the system only decompresses the local blocks that may contain matching data. This approach maintains query accuracy for the relevant portions while significantly reducing memory overhead.
Solution Approach 2:
The patent uses partial action by decompressing only the necessary portion of the compressed data (specific blocks identified by fingerprint matching) rather than the entire dataset. This partial decompression approach ensures that all potentially matching data is processed while avoiding the memory overhead of decompressing irrelevant data blocks.
3Productivity
If fine-granularity block-level pruning is implemented using fingerprints, then query performance is improved, but system complexity increases
Solution Approach 1:
The patent introduces fingerprints as an intermediary structure between the compressed data and the query processing mechanism. These fingerprints act as a mediator that enables efficient block-level pruning without requiring complex decompression logic. The intermediary fingerprints simplify the query process by providing a quick filter mechanism that reduces the amount of data needing full decompression.
Solution Approach 2:
The patent segments the data management system into distinct components: compression with fingerprint generation, fingerprint-based block identification, and selective decompression. This segmentation allows each component to be optimized independently, improving query performance while managing system complexity through modular design.
Data Source
AI summary
The present disclosure involves systems, software, and computer implemented methods for compressed columnar data search using fingerprints. One example method includes compressing columnar data that includes dividing the columnar data into multiple data blocks and generating a fingerprint for each data block, storing the compressed columnar data and the generated fingerprints in an in-memory database, receiving a query for the columnar data, for each in-memory data block stored in the in-memory database, determining whether the in-memory data block satisfies the query and in response to a determination that the in-memory data block does not satisfy the query, pruning the in-memory data block from the multiple data blocks to generate an unpruned set of data blocks, decompressing the unpruned set of data blocks, and performing a query search on the decompressed unpruned set of data blocks for the received query.