Column Store Compression Re-Selection Using Ratio Drift
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems face inefficiencies in re-determining compression algorithms for columnar data, leading to sub-optimal compression ratios and resource wastage, as they rely on manual triggers or row count thresholds, which do not accurately reflect changes in data distribution over time.
Innovation Solution
Calculating and storing compression ratios for each column during initial compression and delta merge operations, using the maximum difference in compression ratios and changed rows to determine when to re-determine suitable compression algorithms, optimizing resource usage and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If periodic re-determination of compression algorithms is performed for each column, then compression ratio improvement is achieved, but resource consumption increases
Solution Approach 1:
The system changes the parameter used to trigger re-determination from row count to compression ratio difference. By monitoring whether the compression ratio deviates from a threshold relative to a baseline, the system intelligently triggers re-compression only when necessary, avoiding unnecessary resource consumption while maintaining compression efficiency.
Solution Approach 2:
The system performs self-monitoring of compression ratios and automatically triggers re-determination when the compression ratio deteriorates beyond a threshold. This self-service mechanism eliminates the need for manual intervention or overly frequent periodic checks, optimizing resource usage.
2Ease of operation
If re-determination is triggered by row count threshold, then simplicity is maintained, but accuracy in detecting compression needs deteriorates
Solution Approach 1:
The triggering parameter is changed from row count to compression ratio difference. This provides a direct and accurate measurement of whether re-compression is needed, replacing the indirect proxy of row count with a precise indicator of compression effectiveness.
Solution Approach 2:
The system replaces the mechanical counting approach (row count thresholds) with a computational measurement approach (compression ratio calculation and comparison). This substitution enables more accurate detection of compression needs while maintaining operational simplicity through automated threshold comparison.
Data Source
AI summary
A system includes application of respective compression types to first data associated with each of a plurality of columns to generate compressed column data, determination of a first compression ratio for each of the plurality of columns based on the compressed column data, storage of the determined first compression ratios, application, for each of the plurality of columns, of the determined compression type to second data associated with the column to generate second compressed column data, determination of a second compression ratio for each of the plurality of columns based on the second compressed column data, determination of a value for each column based on the stored first compression ratio and the second compression ratio determined for the column, determination of a representative value of the determined values, and determination, based on the representative value, whether to re-determine a compression type for each of the plurality of columns.