MPI-IO Collective Buffering Adaptation for High-Speed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern computing systems with high-speed storage targets and interface protocols face performance bottlenecks due to the traditional two-stage collective buffering mechanism in message passing protocols like MPI-IO, which is not adapted to take advantage of faster storage devices.
Innovation Solution
A new access mode is introduced within the MPI-IO protocol that applies a write lock to processes accessing files, disabling collective buffering and allowing each process to write directly to storage, effectively reducing processing overhead by treating all processes as aggregators of their own data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the traditional two-stage collective buffering mechanism is used in MPI-IO, then data consistency and access control are improved, but processing overhead increases and performance bottlenecks occur when accessing high-speed storage devices
Solution Approach 1:
The patent implements dynamic adaptation of the collective buffering mechanism based on storage device type. The system automatically selects between traditional two-stage buffering (for slow storage devices like magnetic disks) and direct write mode (for fast storage devices like SSDs or persistent memory), allowing the access pattern to change dynamically according to the underlying storage capabilities rather than using a fixed approach
Solution Approach 2:
The invention changes the operational parameters of the MPI-IO collective buffering mechanism by introducing a new access mode that modifies the buffering behavior. When accessing high-speed storage devices, the system changes the parameter from traditional two-stage buffering to direct write mode, where all processes write directly to storage without intermediate aggregation, thereby changing the performance characteristics to match the storage device speed
2Reliability
If the traditional two-stage collective buffering mechanism is used, then data consistency is maintained, but the processing overhead and intermediate data collection step create performance bottlenecks with high-speed storage
Solution Approach 1:
The patent extracts and removes the intermediate data collection stage from the traditional two-stage collective buffering mechanism when accessing high-speed storage devices. By taking out the aggregation step where data is collected from multiple processes before writing, the system eliminates the processing overhead and bottleneck associated with this intermediate operation, allowing processes to write directly to storage
3Reliability
If collective buffering is enabled for all processes, then data consistency is ensured, but all processes must participate in data collection which increases processing time for high-speed storage access
Solution Approach 1:
The system dynamically adjusts the collective buffering behavior based on the storage device type. When accessing high-speed storage devices, it transitions to a mode where processes do not participate in intermediate data collection, reducing access time while maintaining data consistency through direct write operations with appropriate locking mechanisms
Data Source
AI summary
An electronic data file management system may comprise storage nodes containing memory devices coupled through a network to compute nodes configured to request group access to a file on the storage nodes, the request being made by at least two processes running on the compute nodes. At least one of the compute nodes is configured to determine the type of a memory device that includes the file, cause the plurality of compute nodes to collectively access the file in a first access mode, applying a write lock mechanism and disabling a collective buffering mechanism to each of the different ones of the compute nodes if the memory device is a first type, and access the file in a second access mode, applying the write lock mechanism and enabling a collective buffering mechanism to each of the different ones of the compute nodes, if the memory device is a second type.