Persistent Partitionable Distributed Map Using Sparse Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance computing applications face inefficiencies in space allocation and performance due to the overhead of checkpoint/restart mechanisms and the limitations of traditional distributed file systems, particularly in achieving both space and performance efficiency using sparse arrays in shared memory for multiprocessing.
Innovation Solution
Implementing a persistent partitionable distributed map (F-Map) using sparse arrays and sparse ordered two-bit bitmaps in shared memory, which allows for efficient space allocation and concurrent access by multiple processes, leveraging Judy arrays and cache coherence to minimize virtual address space usage and optimize read access latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional distributed file systems are used for space allocation, then system compatibility is maintained, but space allocation efficiency and performance deteriorate due to checkpoint/restart overhead
Solution Approach 1:
The patent divides the distributed map into multiple partitions that can be independently managed and accessed. Each partition can be stored in different locations (shared memory, files, databases), allowing parallel operations and eliminating the need for system-wide checkpoint/restart operations. This segmentation enables efficient space allocation without the performance penalty of traditional distributed file systems.
2Quantity of substance
If sparse arrays are used in shared memory for multiprocessing, then space efficiency is improved, but access performance deteriorates due to virtual address space limitations
Solution Approach 1:
The patent transforms the one-dimensional sparse array structure into a multi-dimensional hierarchical structure with partitions and sub-partitions. This dimensional transformation allows the system to bypass virtual address space limitations by distributing data across multiple memory regions and storage locations, while maintaining efficient access patterns through the hierarchical organization. The multi-dimensional structure enables both space efficiency and fast access by allowing parallel operations on different partitions.
3Productivity
If a global map is implemented in shared memory, then multiprocessing efficiency is improved, but memory usage and complexity increase
Solution Approach 1:
The global map is segmented into multiple independent partitions that can be simultaneously accessed by different processes. Each partition is a self-contained unit with its own data structures and access patterns, reducing the complexity of managing a single large global map. This segmentation allows multiple processes to work on different partitions in parallel without requiring complex synchronization mechanisms, thereby improving multiprocessing efficiency while reducing overall system complexity.
4Quantity of substance
If two-bit bitmaps are used instead of traditional bitmaps, then space efficiency is improved, but conversion overhead increases
Solution Approach 1:
The patent pre-organizes the two-bit bitmap data into an optimized format that minimizes conversion overhead during runtime operations. By preparing the data structure in advance with proper indexing and organization, the system can efficiently convert and query the compressed bitmap data without incurring significant performance penalties. This preliminary structuring allows the space-saving benefits of two-bit bitmaps to be realized while maintaining acceptable access speeds.
Data Source
AI summary
One aspect facilitates a global map in a distributed system. The system generates a first data structure which comprises key-value pairs, wherein, in a respective key-value pair, the respective key is an integer and the respective value comprises a pointer to a sparse array which includes a bitmap (such as an ordered two-bit bitmap). The system stores the first data structure as a first partition of the global map. The system searches, based on a pattern, the first data structure to obtain a first value. If the first value comprises a two-bit bitmap, the system converts, based on the pattern, the first value to a two-dimensional bitmap, and performs a function on the first value to obtain a first result. The system uses the first value or the first result as metadata to execute a read or write operation in a filesystem associated with the distributed system.