Compression Hierarchy Paths for Querying Data Without Full Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression schemes require full decompression before query evaluation, leading to high CPU time and memory bandwidth usage, and while techniques allowing SIMD operations on compressed values improve computational load, the remaining computation is still linearly related to the amount of data, limiting efficiency.
Innovation Solution
The implementation of a hierarchical compression system that creates a compression hierarchy of nodes, allowing data to be compressed recursively and selecting a path based on compression statistics to defer or reduce computation during query evaluation, enabling calculations on a subset of compressed values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If full decompression is performed before query evaluation, then data can be operated on, but CPU time and memory bandwidth usage increase significantly
Solution Approach 1:
The patent segments the data processing operation into two distinct phases: a compression phase where data is compressed and statistics are collected, and a query evaluation phase where operations are performed on compressed data using those statistics. This segmentation eliminates the need for full decompression before query evaluation, allowing operations to be performed directly on compressed data representations, thereby reducing CPU time and memory bandwidth usage while maintaining data operability
Solution Approach 2:
The patent performs preliminary compression and statistics collection during the data loading phase, before query evaluation begins. By pre-computing and storing compression statistics (such as value frequencies, ranges, and patterns), the system enables subsequent query operations to be executed on compressed data without requiring full decompression, thus reducing computational overhead during query processing
2Productivity
If SIMD operations are performed on compressed values, then computational load is improved, but computation remains linearly related to data amount
Solution Approach 1:
The patent transitions from operating on individual data values to operating on compressed data representations and their associated statistics. By lifting the operation to a higher dimensional level where compression statistics (aggregated information about data distributions, patterns, and characteristics) are used to guide query evaluation, the system can determine results without processing every individual value, thereby breaking the linear relationship between data size and computation time
Solution Approach 2:
The patent performs partial computation by using compression statistics to identify and process only the relevant portions of compressed data needed to answer a query. Instead of evaluating all compressed values, the system leverages statistical information to prune the search space and focus computational effort only on necessary data segments, reducing computation time below what would be required for complete data evaluation
Data Source
AI summary
According to embodiments of the present invention, machines, systems, methods and computer program products for hierarchical compression of data are presented comprising creating a compression hierarchy of compression nodes, wherein each compression node is associated with a compression operation to produce compressed data. An output of any of the compression nodes may be compressed by another compression node or the same compression node. A path of one or more compression nodes is determined through said compression hierarchy based upon compression statistics to compress data, and the data is compressed by the compression nodes of the path. Various computational techniques are presented herein for manipulating the compression hierarchy to defer or reduce computation during query evaluation.