Dual Storage In-Memory Array and On-Disk Page Structure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face challenges in efficiently managing large data volumes due to the limitations of traditional disk-based storage and in-memory systems, which struggle with random access and scalability, especially as data capacity continues to increase.
Innovation Solution
A dual storage approach using an in-memory array with a contiguous block of memory addresses and an on-disk page structure, where data is encoded in a native format and stored in a page structure that supports quick random access, allowing for efficient loading and storage of data, including the use of a unified table approach that integrates in-memory and on-disk database systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in traditional disk-based storage systems, then data capacity can be increased, but random access performance and system speed deteriorate
Solution Approach 1:
The patent combines in-memory storage and on-disk storage into a unified dual storage system. Data is simultaneously maintained in both in-memory arrays (for fast access) and on-disk page structures (for capacity), with a mapping mechanism that enables fast retrieval from either storage medium, thus achieving both high speed and large capacity
2Speed
If data is stored in in-memory systems, then random access performance is improved, but data capacity and cost-effectiveness worsen
Solution Approach 1:
The system merges in-memory arrays with on-disk page structures, allowing hot data to reside in memory for fast access while cold data is stored on disk. The mapping mechanism enables the system to achieve large data capacity with cost-effective disk storage while maintaining fast random access performance for frequently accessed data through in-memory caching
3Quantity of substance
If traditional disk-based storage is used, then data capacity can be scaled, but system complexity and data management overhead increase
Solution Approach 1:
The patent segments data management into distinct components: in-memory arrays for active data, on-disk page structures for persistent storage, and a mapping mechanism for coordination. This segmentation allows each component to be optimized independently while reducing overall management complexity through clear separation of concerns
4Productivity
If in-memory storage is used, then system performance is improved, but cost per unit storage increases
Solution Approach 1:
The system applies local quality by storing only the most frequently accessed data (hot data) in expensive in-memory storage, while less frequently accessed data (cold data) is stored in cheaper on-disk storage. This localized optimization ensures high performance for critical operations while minimizing overall storage costs
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The subject matter disclosed herein provides methods for the dual storage of data using an in-memory array and an on-disk page structure. An in-memory array holding a column of data can be maintained. One or more pages can be maintained. Each of the one or more pages can have one or more rows for storing the column of data. Random access can be provided to a subset of the one or more rows by at least loading the subset of rows from the one or more pages to the in-memory array without loading all of the rows from the one or more pages. Related apparatus, systems, techniques, and articles are also described.