In-Memory Database Column Encoding and Auxiliary Structures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transition from disk-based to in-memory database query processing poses challenges in minimizing memory and bandwidth usage while maintaining performance, especially in lower-cost and lower-power processing models with numerous processing units.
Innovation Solution
The system adapts data structures and selects optimal auxiliary structures to minimize memory and bandwidth usage by encoding columns based on query operations, using techniques like dictionary encodings, run-length encodings, and B-Trees, and dynamically adjusts encoding during query processing to ensure minimal interference with query operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored in traditional disk-based systems with large main memory, then query processing performance is maintained, but power consumption and cost increase significantly
Solution Approach 1:
The patent segments data into multiple compressed representations (dictionary encoding, run-length encoding, bitmaps, etc.) and stores only the necessary portions in main memory. This allows the system to process queries using fragmented, compressed data segments rather than requiring complete uncompressed data in memory, thereby reducing memory capacity and power consumption while maintaining query performance.
Solution Approach 2:
The system dynamically changes data representation parameters by selecting different encoding methods (dictionary, run-length, bitmap, etc.) based on query workload characteristics. This allows optimal adaptation of data storage format to query patterns, achieving high performance with reduced memory requirements and lower power consumption.
2Quantity of substance
If data is compressed to reduce memory usage, then memory and bandwidth consumption decrease, but query processing complexity increases
Solution Approach 1:
The system performs preliminary compression of data into multiple encoded representations before query processing. By pre-computing and storing dictionary encodings, run-length encodings, and bitmaps, the system eliminates the need for complex runtime compression operations during query execution, thereby reducing operational complexity while maintaining low memory usage.
Solution Approach 2:
The patent introduces intermediary data structures (dictionary tables, run-length encoded arrays, bitmap indices) that mediate between the compressed data and query processing operations. These intermediaries simplify query operations by providing pre-processed, query-friendly representations without requiring complex real-time compression algorithms.
3Productivity
If auxiliary data structures are added to optimize query operations, then query performance improves, but memory consumption increases
Solution Approach 1:
The patent creates multi-functional compressed data structures that serve multiple query operations simultaneously. For example, a single dictionary encoding can support both equality predicates and grouping operations, while bitmaps can support both filtering and aggregation. This universality allows the system to achieve high query performance without proportionally increasing memory consumption, as the same compressed structures fulfill multiple query needs.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for providing optimized data representation of relations for in-memory database query processing is disclosed. The method seeks to optimize the use of the available memory by encoding relations on which the in-memory database query processing is performed and by employing auxiliary structures to maintain performance. Relations are encoded based on data patterns in one or more attribute-columns of the relation and the encoding that is selected is suited to a particular type of data in the column. Members of a set of auxiliary structures are selected based on the benefit the structure can provide and the cost of the structure in terms of the amount of memory used. Encoding of the relations is performed in real-time while query processing occurs, using locks to eliminate conflicts between the query processing and encoding.