Multi-Level Column Compression for Database Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression methods for column storage in databases suffer from low compression efficiency, which results in inefficient use of storage space and suboptimal read performance.
Innovation Solution
A method and apparatus for column data compression that involves encoding column data in multiple schemes at the column level, determining the encoding scheme that occupies the least storage space, and applying a compression algorithm with the highest rate at the stream level to achieve optimal compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If column data is compressed using traditional single-level compression methods, then storage space is reduced, but compression rate remains low
Solution Approach 1:
The patent divides the compression process into two distinct segments: column-level encoding that converts column data into data streams using various encoding schemes (e.g., integer encoding, string encoding, dictionary encoding), and stream-level compression that applies compression algorithms to the encoded data streams. This segmentation allows each level to optimize for its specific function, achieving both high compression rates and effective storage space reduction.
Solution Approach 2:
The patent introduces a hierarchical dimension to the compression process by operating at multiple levels (column level and stream level) rather than a single level. This dimensional approach enables the system to first transform data structure at the column level and then apply compression at the stream level, thereby achieving superior compression rates compared to traditional single-level methods.
2Productivity
If multiple encoding schemes are tested at column level, then compression optimization is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary encoding of column data into multiple data streams using different encoding schemes before the compression stage. By pre-processing the data at the column level and organizing it into standardized data streams, the system reduces the complexity of subsequent compression operations. This preliminary action allows the compression algorithm to work on already-structured data, improving overall optimization while managing computational complexity.
Solution Approach 2:
The patent changes the parameters of data representation by applying different encoding schemes (e.g., converting to integer streams, string streams, or dictionary-encoded streams) at the column level. These parameter changes transform the data into forms that are more amenable to compression, allowing the system to achieve better compression optimization without excessively increasing computational complexity during the actual compression phase.
Data Source
AI summary
This specification provides a column data compression method and apparatus, and a storage medium. The method includes: separately encoding the column data in a plurality of encoding schemes at a column level, to obtain at least one data stream corresponding to each encoding scheme; determining a target encoding scheme in which smallest storage space is occupied in the plurality of encoding schemes; determining a target compression algorithm with a highest compression rate at a stream level; and compressing at least one target data stream corresponding to the target encoding scheme based on the target compression algorithm, and storing compressed target data stream in a database. In the present specification, the column data can be encoded and compressed at at least two levels: the column level and the stream level, to improve a compression rate of the column data, and save more storage space.