Columnar Data Block Multi-Level Compression for Faster Query Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of data being managed by organizations leads to increased complexity and costs in data storage and management, necessitating more efficient solutions to reduce storage requirements and improve data access efficiency.
Innovation Solution
Implementing a multi-level compression scheme in data stores, such as columnar database tables, which applies primary compression techniques based on metadata and further compresses data using secondary, byte-oriented techniques, reducing storage needs and improving query performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored without compression, then data access speed is faster, but storage requirements increase
Solution Approach 1:
The patent applies segmentation by dividing compression into multiple levels: first applying column-specific compression techniques to individual columns, then applying byte-oriented compression to the results. This segmented approach achieves higher overall compression ratios while maintaining query performance through selective decompression of only needed columns.
Solution Approach 2:
The patent implements preliminary action by pre-compressing data using column-specific techniques before storing in the data store. This pre-compression allows the system to achieve smaller storage footprints while enabling efficient query processing through targeted decompression operations.
2Quantity of substance
If multi-level compression is applied, then storage requirements are reduced, but system complexity increases
Solution Approach 1:
The patent uses an intermediary approach by introducing a compression selector component that automatically chooses appropriate compression techniques based on data characteristics. This intermediary layer simplifies the system by abstracting the complexity of multiple compression algorithms behind a unified interface that transparently selects and applies the most suitable technique.
Solution Approach 2:
The patent applies parameter changes by dynamically adjusting compression parameters based on data type, data distribution, and query patterns. The system modifies compression settings such as dictionary size, block size, and compression level to optimize the balance between storage efficiency and query performance for different scenarios.
3Quantity of substance
If column-specific compression is used, then compression efficiency is improved, but metadata management complexity increases
Solution Approach 1:
The patent implements universality by creating a unified metadata structure that serves multiple functions: it stores compression technique identifiers, tracks data distribution statistics, and provides query optimization hints. This multi-functional metadata approach reduces the overhead of managing column-specific compression information while enabling efficient query processing.
Data Source
AI summary
Data to be stored in a data block for a columnar database table may be compressed according to a multi-level compression scheme. Data to be stored in the data block may be received. The data may be compressed according a column-specific compression technique to produce compressed data. The compressed data may then be compressed according to a second compression technique different than the column-specific compression technique to produce multi-level compressed data. The multi-level compressed data may be stored in the data block. When reading from the data block, multi-level compressed data may be decompressed according to the column-specific compression technique and the default compression technique applied to the data.