Hierarchical Data Compression for Query Evaluation Without Full Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression schemes require full decompression before query evaluation, leading to high CPU time and memory bandwidth usage, and while techniques allowing SIMD operations on compressed values reduce computational load, the remaining computation is still linearly related to the amount of data, limiting efficiency.
Innovation Solution
The implementation of a hierarchical compression system that creates a compression hierarchy of nodes, allowing data to be compressed recursively and selecting a path based on compression statistics to defer or reduce computation during query evaluation, enabling calculations on a subset of compressed values without decompressing all data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If full decompression is performed before query evaluation, then data can be operated on, but CPU time and memory bandwidth usage increase significantly
Solution Approach 1:
The patent segments the data processing workflow into distinct compression and query evaluation phases, maintaining compression nodes that represent intermediate compressed states. Query operations are divided into those that can execute on compressed data (filtering, aggregation) versus those requiring decompression, allowing selective processing that avoids full decompression while maintaining data operability through hierarchical compression structures
Solution Approach 2:
The patent performs preliminary compression actions to create hierarchical compression nodes before query evaluation occurs. By pre-computing compression representations and organizing data in a compression hierarchy during data loading, the system prepares the data structure in advance to enable efficient query processing on compressed data without requiring full decompression during query execution
2Productivity
If SIMD operations are performed on compressed values, then computational load is reduced, but computation remains linearly related to data amount
Solution Approach 1:
The patent introduces a hierarchical dimension to compressed data representation, organizing compression nodes in multiple levels where higher-level nodes represent more compressed aggregates. This dimensional transformation allows query operations to traverse the hierarchy and perform computations on aggregated compressed representations rather than processing every individual compressed value linearly, effectively adding a hierarchical dimension to the computation model
3Quantity of substance
If compression hierarchy is created with multiple compression nodes, then compression ratios improve, but system complexity increases
Solution Approach 1:
The patent dynamically adjusts compression parameters and node selection based on data characteristics and query patterns. The system monitors compression ratios, data access patterns, and query types to automatically tune the compression hierarchy depth, node types, and compression algorithms applied at different levels, allowing the system to optimize compression ratios while adapting complexity to actual workload requirements rather than maintaining fixed high complexity
Data Source
AI summary
According to embodiments of the present invention, machines, systems, methods and computer program products for hierarchical compression of data are presented comprising creating a compression hierarchy of compression nodes, wherein each compression node is associated with a compression operation to produce compressed data. An output of any of the compression nodes may be compressed by another compression node or the same compression node. A path of one or more compression nodes is determined through said compression hierarchy based upon compression statistics to compress data, and the data is compressed by the compression nodes of the path. Various computational techniques are presented herein for manipulating the compression hierarchy to defer or reduce computation during query evaluation.