Data Vector Reordering for Compression and Min-Max Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The effectiveness of min-max indexes in database management systems like SAP HANA is hindered by the data order of columns and the compression optimization phase, which can render them ineffective while compromising compression metrics.
Innovation Solution
Adaptive compression optimization is implemented, involving a new reordering strategy that includes sorting data vectors by value and frequency, and prioritizing columns with a certain percentage of distinct values to improve pruning efficiency without sacrificing compression performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If the compression optimization phase reorders data for column compression, then compression metrics are improved, but min-max index effectiveness deteriorates
Solution Approach 1:
The system dynamically adjusts the reordering strategy based on the data characteristics of each column. By calculating the distinct value ratio and selecting different reordering approaches (frequency-based vs. value-based sorting), the system adapts to maintain both compression effectiveness and min-max index performance for different data types
Solution Approach 2:
The invention changes the reordering parameters (sorting criteria) based on the distinct value ratio threshold. When the ratio exceeds the threshold, value-based sorting is applied to preserve min-max index effectiveness; otherwise, frequency-based sorting is used to optimize compression. This parameter change resolves the contradiction by adapting to different data scenarios
2Productivity
If data is reordered for min-max index effectiveness, then pruning performance is improved, but compression metrics deteriorate
Solution Approach 1:
The system applies different reordering strategies to different columns based on their local characteristics (distinct value ratio). Each column is evaluated independently and reordered according to its specific data distribution, ensuring optimal pruning performance for columns that benefit from it while maintaining compression efficiency for others
Solution Approach 2:
The reordering approach is dynamically selected based on the distinct value ratio of each column. The system transitions between frequency-based sorting (for compression) and value-based sorting (for pruning) depending on the data characteristics, allowing both pruning performance and compression metrics to be optimized in their respective appropriate scenarios
3Reliability
If full table scan is used to execute SQL queries, then query completeness is ensured, but scan time increases
Solution Approach 1:
The system performs preliminary reordering of column data during the compression optimization phase to prepare for efficient pruning. By pre-organizing data in a manner that enables effective min-max index usage, the system reduces scan time for subsequent queries while maintaining query completeness through selective block skipping based on min-max comparisons
Data Source
AI summary
A database management system is described that can encode data to generate a plurality of data vectors. The database management system can perform the encoding by using a dictionary. The database management system can adaptively reorder the plurality of data vectors to prepare for compression of the plurality of data vectors. During a forward pass of the adaptive reordering, most frequent values of a data vector of the plurality of data vectors can be moved-up in the data vector. During a backward pass of the adaptive reordering, content within a rest range of a plurality of rest ranges can be rearranged within the plurality of data vectors according to frequencies of the content. The reordering according to frequency can further sort the rest range by value. Related apparatuses, systems, methods, techniques, computer programmable products, computer readable media, and articles are also described.