Adaptive Dictionary Compression for Column-Store Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Column-store databases face challenges in managing memory usage for string dictionaries, leading to delays in access operations due to high memory consumption, especially when column data exceeds available memory, causing data to be unloaded from memory and reloaded from disk, which is inefficient.
Innovation Solution
A compression manager is implemented to select and apply various dictionary compression variants, such as Huffman coding, Hu-Tucker coding, front coding, bit compression, N-gram compression, Re-Pair compression, and column-wise bit compression, based on evaluation of compressed dictionary size and run-time performance, to optimize memory usage and access speeds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If string dictionaries are stored in memory for fast access, then access speed is improved, but memory consumption increases causing data to be unloaded to disk
Solution Approach 1:
The patent applies multiple dictionary compression variants (Huffman coding, Hu-Tucker coding, front coding, bit compression, N-gram compression, Re-Pair compression, and column-wise bit compression) that transform the storage parameters of string dictionaries. By changing the encoding parameters from uncompressed storage to various compressed formats, the patent reduces memory consumption while maintaining acceptable access speeds, thereby resolving the contradiction between fast access and memory usage.
Solution Approach 2:
The patent implements a dynamic compression manager that selects and switches between different compression variants based on runtime conditions such as memory availability and access patterns. This dynamic adaptation allows the system to optimize the balance between access speed and memory consumption in real-time, resolving the contradiction by making the storage format flexible rather than static.
2Quantity of substance
If column data exceeds available memory, then more data can be stored, but access operations become inefficient due to disk I/O
Solution Approach 1:
By applying compression variants to string dictionaries, the patent changes the storage parameters to reduce the space required for column data. This allows more columns to be kept in memory within the same memory capacity, eliminating the need for disk I/O and maintaining access operation efficiency while increasing storage capacity.
Solution Approach 2:
The patent segments the data storage into compressed dictionary portions and column vector portions, allowing selective compression of string dictionaries while keeping column vectors in memory. This segmentation enables efficient use of memory resources and reduces disk I/O operations.
3Quantity of substance
If multiple compression variants are evaluated and selected, then optimal compression is achieved, but system complexity increases
Solution Approach 1:
The compression manager dynamically selects from multiple compression variants based on runtime conditions, but the selection logic is designed to manage complexity by evaluating variants systematically and choosing the most appropriate one for current conditions, thereby achieving optimal compression without overwhelming system complexity.
Solution Approach 2:
The system performs self-service by automatically evaluating and selecting compression variants without requiring manual intervention. The compression manager autonomously monitors memory usage and access patterns, selecting appropriate compression methods, which manages complexity through automation rather than manual configuration.
Data Source
AI summary
Innovations for adaptive compression and decompression for dictionaries of a column-store database can reduce the amount of memory used for columns of the database, allowing a system to keep column data in memory for more columns, while delays for access operations remain acceptable. For example, dictionary compression variants use different compression techniques and implementation options, Some dictionary compression variants provide more aggressive compression (reduced memory consumption) but result in slower run-time performance. Other dictionary compression variants provide less aggressive compression (higher memory consumption) but support faster run-time performance. As another example, a compression manager can automatically select a dictionary compression variant for a given column in a column-store database. For different dictionary compression variants, the compression manager predicts run-time performance and compressed dictionary size, given the values of the column, and selects one of the dictionary compression variants.