XVM Hierarchical Maps for Parallel Storage Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional virtual memory systems suffer from slow read/write speeds and inefficiencies in managing large data sets, leading to performance drops and 'thrashing' when systems rely heavily on hard drive storage, especially in applications requiring simultaneous access to multiple programs.
Innovation Solution
The eXtreme Virtual Memory (XVM) programming model provides a hierarchical data abstraction for efficient data distribution and management across storage hierarchies, using hierarchical maps to partition data into subarrays and manage data movement between storage levels, allowing for dynamic data movement policies and a global view of data across multiple processors and storage hierarchies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional virtual memory systems use hard drive storage for overflow memory, then memory capacity is increased, but read/write speed deteriorates significantly
Solution Approach 1:
The patent segments the storage system into multiple tiers with different performance characteristics (fast memory, slower memory, disk storage). The hierarchical map divides the global address space into segments that can be mapped to different storage tiers, allowing frequently accessed data to reside in faster memory while less frequently accessed data is stored on disk, thus resolving the contradiction between capacity and speed.
Solution Approach 2:
The patent introduces a new dimension of organization through the hierarchical map structure, which adds layers of abstraction between the logical address space and physical storage. This hierarchical mapping mechanism enables the system to manage data across multiple storage dimensions (speed, capacity, cost) simultaneously, allowing the system to achieve both large capacity and acceptable speed through intelligent data placement.
2Ease of manufacture
If the operating system uses general-purpose paging algorithms, then implementation simplicity is improved, but performance for specific applications deteriorates
Solution Approach 1:
The patent implements dynamic data movement policies where the hierarchical map can be reconfigured based on application requirements and access patterns. The system dynamically adjusts which data segments are placed in which storage tiers, allowing optimization for specific applications while maintaining a general-purpose framework. This dynamic adaptation resolves the contradiction between implementation simplicity and application-specific performance.
3Adaptability or versatility
If the system constantly swaps pages between memory and disk, then memory management flexibility is improved, but system performance deteriorates due to thrashing
Solution Approach 1:
The patent employs preliminary action by pre-configuring the hierarchical map to anticipate data access patterns and proactively placing data in appropriate storage tiers before access occurs. The system performs preliminary data movement based on metadata and access history, reducing the need for constant swapping during operation. This prevents thrashing while maintaining memory management flexibility.
Solution Approach 2:
The patent implements feedback mechanisms that monitor data access patterns and use this information to dynamically adjust the hierarchical map configuration. The system continuously gathers feedback on which data is frequently accessed and reallocates storage resources accordingly, reducing unnecessary page swaps. This feedback-driven approach maintains flexibility while improving performance by minimizing thrashing.
Data Source
AI summary
A method and computer program product for orchestrating the distribution and management of parallel data on a parallel hierarchical storage system is presented. A hierarchical map of the parallel data is provided. The hierarchical map comprises a plurality of map objects, wherein each map object describes how to partition a given array of data into a plurality of subarrays of data created by a parent map object of the map object and how to store the parallel data into the parallel computer's storage hierarchy.