Memory-Mapped Data Cube Segmentation for High-Throughput Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern enterprise computing systems face high latency and low throughput when accessing large data sets, leading to delays in query results and document generation, due to the need for extensive memory allocation and storage for data cubes, which often exceed the available system memory.
Innovation Solution
Implementing memory-mapped storage for data cubes, where frequently accessed segments are stored in RAM and less frequently accessed segments are stored on disk, using managed buffers to reduce memory allocations and optimize storage, allowing for efficient caching and dynamic adjustment based on access patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data cubes are stored entirely in system memory (RAM), then data access speed and throughput are improved, but system memory requirements increase dramatically
Solution Approach 1:
The data cube is divided into multiple segments that can be independently managed and stored. Frequently accessed segments are kept in RAM while less frequently accessed segments are stored on disk, allowing the system to maintain high access speeds for hot data while avoiding the need to load the entire data cube into memory
Solution Approach 2:
A memory mapping mechanism serves as an intermediary layer between the application and the physical storage system. This layer transparently manages data movement between RAM and disk, providing fast access to frequently used data while automatically handling less frequently accessed data on slower storage media
2Quantity of substance
If memory-mapped files are used for data cube storage, then memory overhead is reduced, but access performance may deteriorate due to disk I/O
Solution Approach 1:
The system dynamically adjusts which data segments are kept in memory based on access patterns. The memory mapping mechanism continuously monitors usage and automatically moves frequently accessed segments into RAM while leaving less frequently accessed segments on disk, optimizing the balance between memory overhead and access performance
Solution Approach 2:
The system changes the storage location parameter for different data segments based on their access characteristics. By dynamically adjusting where data is stored (RAM vs. disk) according to access frequency and recency, the system achieves low memory overhead while maintaining high access speeds for relevant data
3Adaptability or versatility
If multiple large data cubes are maintained concurrently, then system versatility and data availability are improved, but total memory requirements exceed available system memory
Solution Approach 1:
Each data cube is segmented and only the actively used portions are loaded into memory. The memory mapping mechanism allows multiple large data cubes to be represented in memory through pointers and virtual addresses, with actual data residing on disk until needed
Solution Approach 2:
The memory mapping infrastructure provides a universal mechanism that works for all data cubes regardless of their size or content. This single system serves multiple functions: managing individual data cubes, handling multiple concurrent data cubes, and adapting to different access patterns across all cubes
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for memory mapping to enhance data cube performance. In some implementations, a system accesses a data set that includes data to be processed into a data cube. The system generates a memory-mapped data cube that includes a plurality of files including different segments of the data cube. Generating the memory-mapped data cube includes allocating memory-mapped buffers in non-volatile data storage and responding to subsequent memory allocation requests with addresses for the buffers such that components of the data cube are accumulated in the buffers. The memory-mapped data cube is loaded by storing the files of the data cube in disk-based storage, mapping the stored files of the data cube to virtual memory addresses, and caching portions of the data cube in random-access memory.