Selective Column Compression for Faster Access to OLAP Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques for large volumes of data, such as those used in OLAP database applications, face challenges in efficiently compressing and accessing data, particularly in column-store databases, where locating specific data values quickly is difficult due to large data volumes and inefficient compression methods.
Innovation Solution
The approach involves selectively compressing specific subsets of data, like columns in a table, using techniques like run-length encoding and dictionary compression, while leaving other subsets uncompressed, and reordering data to improve compression efficiency. A data structure is provided to facilitate quick access to compressed data, using methods like run-store and location-store formats for run-length encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression techniques are applied to all data subsets, then storage space is reduced, but data retrieval speed deteriorates
Solution Approach 1:
The patent divides the data set into multiple proper data subsets and selectively applies compression to only those subsets that meet specific criteria (e.g., high compressibility, frequent access patterns). This segmentation allows the system to maintain fast access to uncompressed frequently-accessed data while achieving storage reduction on less frequently accessed data.
Solution Approach 2:
Different compression techniques and strategies are applied to different data subsets based on their specific characteristics. The patent evaluates each subset's compressibility and access patterns to determine the optimal compression approach for that local region of data, rather than applying a uniform compression strategy across all data.
2Quantity of substance
If compression techniques are applied to reduce storage space, then storage efficiency is improved, but complexity of data management increases
Solution Approach 1:
The patent performs preliminary evaluation of data subsets to identify which ones are suitable for compression before actually applying compression techniques. This preliminary action includes assessing compressibility metrics and access patterns, allowing the system to make informed decisions about compression strategy and avoid unnecessary compression of data that wouldn't benefit from it.
Solution Approach 2:
The patent introduces intermediate data structures and indexing mechanisms that facilitate management of compressed data without requiring complex decompression and recompression operations. These intermediaries act as mediators between the compressed data storage and the access operations, simplifying the overall data management complexity.
3Quantity of substance
If data is reordered to improve compression, then compression efficiency is improved, but data access time increases
Solution Approach 1:
The patent reorders data within individual proper data subsets independently rather than reordering the entire data set. This segmentation of reordering operations allows compression improvement within each subset while maintaining the ability to access data in original row order through the columnar structure.
Solution Approach 2:
The patent maintains the columnar storage structure which allows data to be physically reordered for compression purposes while logically preserving the row-based access pattern. This dimensional separation between physical storage order and logical access order enables both improved compression and maintained access performance.
Data Source
AI summary
Techniques and solutions are described for compressing data and facilitating access to compressed data. Compression can be applied to proper data subsets of a data set, such as to columns of a table. Using various methods, the proper data subsets can be evaluated to be included in a group of proper data subsets to be compressed using a first compression technique, where unselected proper data subsets are not compressed using the first compression technique. Data in the data set can be reordered based on a reordering sequence for the proper data subsets. Reordering data in the data set can improve compression when at least a portion of the proper data subsets are compressed. A data structure is provided that facilitates accessing specified data stored in a compressed format.