Intermediate Data Migration Handling System for HPC Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance computer clusters face inefficiencies in data migration due to random I/O activity from data generating entities, leading to prolonged I/O cycles and increased power consumption, while maintaining data integrity and reliability remains a challenge.
Innovation Solution
An intermediate data migration handling system decouples random I/O activity by employing an intermediate data storage layer for caching and organizing data transfers, performing RAID calculations for reliability, and optimizing data transfer sequences to minimize power consumption and maximize bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data generating entities perform random I/O activity directly to data storage disks, then data access flexibility is improved, but I/O cycle duration increases and power consumption increases
Solution Approach 1:
The patent introduces an intermediate data storage layer (cache) between data generating entities and data storage disks. This intermediary buffer decouples the random I/O requests from the sequential storage operations, allowing flexible data access while maintaining efficient disk utilization and reducing I/O cycle duration.
Solution Approach 2:
The patent segments the data migration process into two distinct phases: (1) caching random I/O data in the intermediate storage layer, and (2) sequentially migrating cached data to the data storage disks. This segmentation allows each phase to be optimized independently, improving overall I/O efficiency.
2Ease of operation
If data generating entities perform random I/O activity directly to data storage disks, then data access flexibility is improved, but power consumption increases
Solution Approach 1:
The intermediate data storage layer acts as a buffer that absorbs random I/O writes without immediately accessing the power-consuming disk subsystem. By batching and sequencing disk operations, the system reduces the frequency and duration of disk activations, thereby lowering overall power consumption while maintaining data access flexibility.
Solution Approach 2:
The system performs preliminary caching of data in the intermediate storage layer before committing to disk storage. This preliminary action allows the system to prepare data in an optimized format and sequence, reducing the need for repeated disk accesses and minimizing power consumption.
3Loss of time
If data is cached in an intermediate storage layer before migrating to target storage, then I/O cycle duration is reduced, but system complexity increases
Solution Approach 1:
The patent introduces an intermediate data storage layer (cache) between data generating entities and data storage disks. This intermediary buffer decouples the random I/O requests from the sequential storage operations, allowing flexible data access while maintaining efficient disk utilization and reducing I/O cycle duration.
Solution Approach 2:
The intermediate data storage layer serves multiple functions: (1) buffering random I/O requests, (2) sequencing disk operations, (3) enabling early compute state resumption, and (4) simplifying the interface between data generators and storage disks. This multi-functionality justifies the added complexity by delivering comprehensive performance benefits.
4Reliability
If RAID calculations are performed on ingress data before storing, then data reliability is improved, but processing time increases
Solution Approach 1:
The system performs RAID calculations (parity generation, data validation) on ingress data while it is being cached in the intermediate storage layer, before the data is committed to the target storage disks. This preliminary processing ensures data reliability is established early, and the results can be reused during subsequent read operations, reducing overall processing time.
Solution Approach 2:
The RAID calculations are performed continuously as data arrives at the intermediate storage layer, rather than as a separate batch operation. This continuous processing overlaps with the caching operation, ensuring that data reliability checks are completed without adding significant overhead to the overall I/O cycle.
Data Source
AI summary
A system for data migration between high performance computing architectures and data storage disks includes an intermediate data migration handling system which has an intermediate data storage module coupled to the computer architecture to store data received, and a data controller module which includes data management software supporting the data transfer activity between the intermediate data storage module and the disk drives in an orderly manner independent of the random I/O activity of the computer architecture. RAID calculations are performed on the data prior to storage in the intermediate storage module, as well as when reading data from it for assuring data integrity, and carrying out reconstruction of corrupted data. The data transfer to the disk drives is actuated in sequence determined by the data management software based on minimization of seeking time, tier usage, predetermined time since the previous I/O cycle, or fullness of the intermediate data storage module. The storage controller deactivates the disk drives which are not needed for the data transfer.