SSD Burst Buffer for Supercomputer I/O Bottleneck
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current supercomputing storage architectures face challenges in achieving balanced bandwidth and capacity, particularly as the trend shifts towards more cores with smaller memories per core, leading to random access patterns that decrease disk throughput and increase the need for a large number of disks, making the system unaffordable and inefficient.
Innovation Solution
The introduction of a Solid-State Drive (SSD) burst buffer node interposed between compute nodes and magnetic disk storage, combined with the Parallel Log-Structured File System (PLFS), transforms random access patterns into sequential ones, reducing the number of disks required and improving I/O performance by optimizing data layout and access patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If the system uses more cores with smaller memories per core, then computational power is improved, but random access patterns decrease disk throughput and increase the number of disks required
Solution Approach 1:
The patent introduces an SSD burst buffer as an intermediary component between compute nodes and magnetic disk storage. This SSD burst buffer acts as a mediator that receives random write requests from multiple compute nodes, buffers the data, and then performs sequential writes to the magnetic disk storage, thereby decoupling the random access patterns from the disk subsystem and restoring high throughput.
Solution Approach 2:
The patent segments the storage system into two distinct layers: an SSD burst buffer layer that handles random write operations from compute nodes, and a magnetic disk storage layer that handles sequential writes. This segmentation allows each layer to be optimized for its specific access pattern, with SSDs handling random writes efficiently and magnetic disks handling sequential writes at high throughput.
2Power
If the system uses more cores with smaller memories per core, then computational power is improved, but the number of disks required increases making the system unaffordable
Solution Approach 1:
The SSD burst buffer serves as an intermediary that aggregates random write requests from multiple compute nodes into a single sequential write stream to the magnetic disk storage. This consolidation allows a small number of magnetic disks to serve many compute nodes, dramatically reducing the total number of disks required while maintaining high computational power.
Solution Approach 2:
The patent combines multiple random write requests from different compute nodes into a single sequential write operation to the magnetic disk storage. By merging these operations through the SSD burst buffer, the system achieves high computational power with a reduced number of disks, as the magnetic disks are utilized more efficiently.
3Device complexity
If the system uses traditional direct connection between compute nodes and magnetic disk storage, then system complexity is lower, but I/O performance and bandwidth are insufficient
Solution Approach 1:
The patent introduces an SSD burst buffer node as an intermediary component that significantly enhances I/O throughput. Although this adds a component to the system, the SSD burst buffer provides substantial performance benefits by buffering random writes from compute nodes and performing sequential writes to magnetic disk storage, achieving high I/O throughput that justifies the added complexity.
Data Source
AI summary
For simulating a parallel supercomputing cluster, a simulation program includes a model of the system of compute nodes and a model of an I/O subsystem that stores checkpoints from the compute nodes. When executed, instructions in the simulation program perform the steps of receiving input parameters defining the compute nodes and the I/O subsystem, computing a total number of computational flops for a time between checkpoints and an amount of disk storage required to store the checkpoint data for different configurations of the parallel supercomputing cluster, and presenting a summary to a user of the computed number of computational flops for a time between checkpoints and an amount of disk storage required to store the checkpoint data for the different configurations of the parallel supercomputing cluster.