Fixed String Dictionary for Column-Based In-Memory Database
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional row-based databases face challenges in minimizing memory consumption for efficient data access, as each row requires fixed storage space regardless of identical values, leading to increased memory usage and slower access times.
Innovation Solution
Implementing a column-based in-memory database system with a fixed string dictionary where each column has a dictionary of distinct values, and each row stores a reference to these values, allowing for compression and efficient storage through bitvector and bitarray encoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If row-based storage is used with fixed allocation per row, then data access is straightforward, but memory consumption increases significantly
Solution Approach 1:
The patent segments data storage into two distinct components: a dictionary structure that stores unique values, and a reference structure that stores only the position indices. This segmentation allows the system to separate the storage of actual data values from the storage of references, thereby reducing overall memory consumption while maintaining efficient access patterns.
Solution Approach 2:
Instead of storing actual data values in each row, the patent creates a copy of the data in dictionary form and stores only references (indices) to these copies. This copying approach eliminates redundant storage of identical values across multiple rows, significantly reducing memory usage while preserving the ability to retrieve original data through the dictionary lookup.
2Quantity of substance
If all distinct values are stored in a dictionary for each column, then memory usage is reduced, but access time may increase due to additional lookup operations
Solution Approach 1:
The patent performs preliminary organization of data into a dictionary structure during data loading or preprocessing. By pre-sorting and deduplicating values into the dictionary before actual access occurs, the system eliminates the need for runtime search operations, allowing direct index-based access to unique values and thus preventing access time penalties.
Solution Approach 2:
The patent transforms the one-dimensional row-based access pattern into a two-dimensional access pattern involving dictionary lookup and reference following. This dimensional change allows the system to trade storage space for access efficiency by organizing data in a way that enables rapid retrieval through index-based direct access rather than sequential scanning.
Data Source
AI summary
The subject matter described herein relates to implementation of a dictionary in a column-based, in-memory database where values are not stored directly, rather, for each column, a dictionary is created with all distinct values. For each row, a reference to the corresponding value in the dictionary is stored. In one aspect, data is stored in a memory structure organized in a column store format defined by a plurality of columns and a plurality of rows. A dictionary for each column in the memory structure is generated. The dictionary has distinct values for each column. A reference to the dictionary is generated for each column in the memory structure. The dictionary and the reference to the dictionary are stored in the memory structure.