Main Memory Database Page Index Vectors for Large Dataset Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional main memory databases struggle to handle extremely large datasets, resulting in inadequate processing speed and predictability, limiting their ability to meet user expectations for data analysis.
Innovation Solution
A system utilizing an index vector with a compression manager and page generator to store and load pages of value identifiers from secondary memory into main memory, employing dictionary and n-bit compression to ensure fast and reliable data access, and implementing on-demand paging to manage large data volumes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the entire index vector is loaded into main memory, then data access speed is improved, but memory requirements exceed available main memory capacity
Solution Approach 1:
The index vector is divided into multiple fixed-size pages stored in secondary memory. Only the required page containing the requested value ID is loaded into main memory, rather than loading the entire index vector. This segmentation allows fast access within the loaded page while keeping overall memory requirements manageable.
Solution Approach 2:
Pages of the index vector are pre-organized and stored in secondary memory in a structured format that enables efficient retrieval. When a value ID is requested, the system can quickly determine which page contains the value and load only that specific page into main memory before access, minimizing wait time.
2Quantity of substance
If compression is applied to reduce memory usage, then storage capacity is improved, but access complexity increases
Solution Approach 1:
Instead of storing the actual large values, the system stores compressed value IDs that reference the original values. Each value ID is a compact representation that can be quickly accessed and then used to retrieve or reference the corresponding full value, significantly reducing storage requirements while maintaining fast access through the compact ID structure.
3Quantity of substance
If the index vector is stored in secondary memory, then memory efficiency is improved, but access time increases due to loading delays
Solution Approach 1:
By dividing the index vector into fixed-size pages, the system minimizes the amount of data that needs to be transferred from secondary to main memory. Only the specific page containing the requested value ID is loaded, reducing transfer time and improving access efficiency while maintaining high memory efficiency.
Solution Approach 2:
The index vector pages are pre-structured and stored in secondary memory with metadata that enables rapid identification of the containing page for any given value ID. This preliminary organization allows the system to quickly locate and load the required page without scanning the entire index vector, minimizing access time despite secondary memory storage.
Data Source
AI summary
A compression manager may store, within an index vector, a plurality of value identifiers (IDs), each value ID representing a value within a database. A page generator may designate a number of the value IDs as defining a page within the index vector, so that the index vector includes a plurality of pages, each page including the number of value IDs. The page generator may store the index vector in a secondary memory of a main memory database. An iterator may access a requested value ID, and a page loader may load a corresponding page of the index vector that contains the requested value ID into the main memory database.