Shared Cache Zero-Copy Memory-Mapped Database
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analysis systems face limitations in scalability and performance due to the need for each application to load and manage a large common data set independently, leading to increased system latency and reduced concurrency.
Innovation Solution
Implementing a shared cache as a zero-copy memory-mapped database, where a single instance of the data set is stored and multiple applications can access it concurrently, arranged in a column store for efficient data access and aggregation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If each application loads the entire data set into its own memory address space, then the application can access data quickly, but the system can support fewer concurrent applications and uses more memory resources
Solution Approach 1:
The patent merges the data storage into a single shared memory space that multiple applications can access simultaneously. Instead of each application having its own copy of the data set, a single instance of the data is loaded into shared memory, allowing multiple applications to reference the same data without duplicating it, thereby supporting more concurrent applications while maintaining fast access speeds.
Solution Approach 2:
The patent introduces a shared memory space as an intermediary between the data set and multiple applications. This shared memory acts as a mediator that allows applications to access data without each application needing its own dedicated memory space, resolving the conflict between fast data access and supporting multiple concurrent applications.
2Ease of operation
If the data set is stored in a traditional row-based format, then general data access is simple, but data aggregation operations are inefficient
Solution Approach 1:
The patent segments the data storage into column-based units rather than traditional row-based structures. By organizing data into separate columns for different attributes, the system enables efficient aggregation operations on specific columns while maintaining simple access patterns. Each column can be independently accessed and processed, improving aggregation productivity without sacrificing access simplicity.
Solution Approach 2:
The patent applies different data organization qualities to different parts of the data structure. While the overall structure supports simple access, specific columns are optimized for aggregation operations. This local optimization allows the system to maintain ease of operation for general access while achieving high productivity for aggregation tasks on specific data columns.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A technique for concurrently accessing a data set includes initializing a shared cache with a column data store configured to store an expected data set in columns and creating a memory map for accessing the physical memory location in the shared cache. Other operations include mapping the applications' data access requests to the shared cache with the memory map. One advantage of the disclosed technique is that only one instance of the expected data set is stored in memory, so each application is not required to create additional instances of the expected data set in the applications memory address space. Therefore, larger expected data sets may be entirely stored in memory without limiting the number of applications running concurrently.