In-Memory Columnar Database Compression Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression systems in in-memory database management systems face challenges in improving search response time and optimizing overall database table compression, leading to inefficiencies in memory consumption.
Innovation Solution
The implementation of a process that computes an aggregated compression score for in-memory columnar database tables by prioritizing and grouping records based on the maximum number of occurrences of value-ids, using a combination of prefix length encoding and greedy approaches to optimize memory usage, and applying a branch and bound approach to prune unnecessary nodes in the compression tree.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data compression systems are applied to in-memory database tables, then memory consumption is reduced, but search response time increases
Solution Approach 1:
The patent segments the compression process by applying different compression techniques to different columns based on their characteristics. It divides columns into groups (e.g., high-cardinality, low-cardinality, numeric, string) and applies appropriate compression algorithms to each group, thereby optimizing both compression ratio and search performance for different data types simultaneously
Solution Approach 2:
The patent implements dynamic compression by allowing the compression technique for each column to be adjusted based on access patterns and data characteristics. The system can dynamically switch between different compression methods or adjust compression parameters to balance memory usage and search response time based on actual workload requirements
2Quantity of substance
If multiple data compression systems are applied to compress data, then memory consumption is reduced, but overall database table compression optimization becomes challenging
Solution Approach 1:
The patent creates a universal compression framework that handles multiple compression techniques through a single standardized interface. The system provides unified column grouping logic, standardized compression parameters, and consistent performance metrics across different compression methods, thereby simplifying the management of multiple compression systems while maintaining their individual benefits
Solution Approach 2:
The patent optimizes compression by adjusting parameters such as compression ratio, block size, and algorithm selection based on column characteristics and performance requirements. It uses parameter tuning to achieve optimal balance between memory consumption and search performance for each column type without requiring complex system architecture
Data Source
AI summary
Various embodiments of systems and methods for optimizing in-memory column based database table compression. A system may receive a parsed in-memory columnar database table as an input and reads an aggregated compression score from memory corresponding to the parsed database table. A total compression score may be computed iteratively for the database table in order to determine whether the total compression score is greater than the aggregated compression score. When the total compression score determined to be greater than the aggregated compression score, the aggregated compression score is assigned to a best compression score. To optimize in-memory database table compression, the given database table may be compressed using the best compression score.