Adaptive Data Vector Reordering for Compression and MMI Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The effectiveness of min-max indexes (MMIs) in database management systems like SAP HANA is compromised by the compression optimization phase, which reorders data for optimal compression, thereby rendering MMIs ineffective in most cases.
Innovation Solution
Adaptive compression optimization is employed, involving a new reordering strategy and selection order for column/row candidates, which includes sorting data vectors by value and frequency, and concatenating rest ranges to improve the pruning efficiency of MMIs without sacrificing compression metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If data is reordered for compression optimization, then compression ratio is improved, but min-max index effectiveness deteriorates
Solution Approach 1:
The data is divided into prefix part and non-prefix part, with min-max index applied only to the non-prefix part. This segmentation allows compression optimization on the prefix while preserving MMI effectiveness on the non-prefix portion.
Solution Approach 2:
Different parts of the data are treated differently: the prefix part is optimized for compression, while the non-prefix part maintains data ordering for MMI effectiveness. This local differentiation resolves the contradiction between compression and index effectiveness.
2Loss of substance
If traditional compression optimization is applied, then compression size is reduced, but scan time increases
Solution Approach 1:
The sorting strategy is changed from frequency-based to value-based sorting, and rest ranges are concatenated to improve data locality. These parameter changes enable both good compression and fast scan performance by preserving spatial proximity of related data.
Solution Approach 2:
The reordering strategy is made adaptive based on data characteristics. By analyzing data patterns and dynamically choosing sorting approaches (value-based vs frequency-based) and rest range concatenation, the system optimizes for both compression and scan performance based on actual data properties.
Data Source
AI summary
A database management system is described that can encode data to generate a plurality of data vectors. The database management system can perform the encoding by using a dictionary. The database management system can adaptively reorder the plurality of data vectors to prepare for compression of the plurality of data vectors. During a forward pass of the adaptive reordering, most frequent values of a data vector of the plurality of data vectors can be moved-up in the data vector. During a backward pass of the adaptive reordering, content within a rest range of a plurality of rest ranges can be rearranged within the plurality of data vectors according to frequencies of the content. The reordering according to frequency can further sort the rest range by value. Related apparatuses, systems, methods, techniques, computer programmable products, computer readable media, and articles are also described.