Column Data Compression with Selective Encoding and Fast Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques for large volumes of data, such as those used in OLAP database applications, face challenges in efficiently compressing and accessing data, particularly in column-store databases, where locating specific data values quickly is difficult due to large data volumes and inefficient compression methods.
Innovation Solution
The approach involves selectively compressing specific subsets of data, like columns in a table, using techniques like run-length encoding and dictionary compression, while leaving other subsets uncompressed, and reordering data to improve compression efficiency. A data structure is provided to facilitate quick access to compressed data, using methods like run-store and location-store formats for run-length encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression techniques are applied to all columns in a column-store database, then storage space is reduced, but data retrieval speed deteriorates due to increased complexity in locating specific data values
Solution Approach 1:
The patent segments the columns into two distinct groups: compressed columns and uncompressed columns. This segmentation allows the system to apply compression selectively rather than universally, thereby reducing storage space for suitable columns while maintaining fast access paths for columns that require quick retrieval, thus resolving the contradiction between storage efficiency and retrieval speed.
Solution Approach 2:
The patent applies different quality characteristics to different parts of the data structure by compressing only those columns where compression is beneficial (reducing storage) while leaving other columns uncompressed (maintaining access speed). This local differentiation of compression quality allows optimization of storage space without uniformly degrading retrieval performance across all columns.
2Quantity of substance
If compression techniques are applied to reduce storage space, then storage efficiency improves, but the complexity of managing and accessing compressed data increases
Solution Approach 1:
By segmenting columns into compressed and uncompressed groups, the patent simplifies the management complexity compared to applying a single complex compression scheme to all columns. The system only needs to manage compression metadata for the compressed subset, reducing the overall management overhead while still achieving storage efficiency gains.
3Quantity of substance
If data is reordered to improve compression efficiency, then compression ratio improves, but the time required for data processing increases
Solution Approach 1:
The patent performs data reordering as a preliminary action during the data loading or ETL phase, before the compression is applied. By reordering the data beforehand to group similar values together, the system maximizes compression efficiency without needing to repeatedly reorder data during subsequent access operations, thus minimizing the time loss impact on regular data processing operations.
Data Source
AI summary
Techniques and solutions are described for compressing data and facilitating access to compressed data. Compression can be applied to proper data subsets of a data set, such as to columns of a table. Using various methods, the proper data subsets can be evaluated to be included in a group of proper data subsets to be compressed using a first compression technique, where unselected proper data subsets are not compressed using the first compression technique. Data in the data set can be reordered based on a reordering sequence for the proper data subsets. Reordering data in the data set can improve compression when at least a portion of the proper data subsets are compressed. A data structure is provided that facilitates accessing specified data stored in a compressed format.