Parallel Checksumming for Silent Data Corruption in HPC Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High Performance Computing (HPC) environments face challenges in ensuring end-to-end data integrity due to the increasing likelihood of silent data corruption as scale grows, with existing checksum methods having a small probability of failure and not detecting corruption effectively.
Innovation Solution
Implementing a distributed data storage system with Parallel Log-Structured File System (PLFS) and key-value storage elements that generate and store checksums for data integrity, ensuring all layers of the software stack verify data integrity through checksums, and using an I/O Dispatcher to manage the burst buffer tier for network-attached flash storage and asynchronous data migration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed storage systems are scaled to exascale environments, then storage capacity and processing power increase, but the probability of silent data corruption increases
Solution Approach 1:
The system performs preliminary checksumming at the time of data writing to storage. Checksum values are computed and stored alongside the data before any potential corruption can occur, enabling future verification without requiring re-computation or external validation.
Solution Approach 2:
The system implements feedback mechanisms where checksums are verified when data is read from storage. If verification fails indicating corruption, error messages are raised and the application is notified, creating a closed-loop feedback system that detects and reports data integrity issues.
2Reliability
If traditional checksumming methods are used, then data integrity verification is provided, but the probability of detection failure remains high at scale
Solution Approach 1:
The system changes the parameter of checksum computation by processing data in parallel chunks with individual checksums, then combining these checksums using associative properties. This mathematical transformation maintains verification reliability while enabling parallel processing that scales efficiently to exascale environments.
Solution Approach 2:
The system segments large datasets into smaller chunks, computing checksums for each chunk independently in parallel. This segmentation enables both improved detection coverage across the entire dataset and efficient parallel processing, addressing both reliability and performance requirements.
3Reliability
If parallel checksumming is implemented across distributed storage, then data integrity is improved, but system complexity increases
Solution Approach 1:
The storage system performs self-verification through automated checksum validation when data is read. The system serves its own integrity verification needs without requiring external validation tools or manual intervention, reducing operational complexity while maintaining high reliability.
Solution Approach 2:
The checksumming mechanism serves multiple functions: it provides data integrity verification, enables parallel processing optimization, and supports error detection across all storage operations. This multi-functionality reduces the need for separate verification systems, thereby managing complexity.
Data Source
AI summary
End-to-end data integrity is provided in parallel computing systems, such as High Performance Computing (HPC) environments. An exemplary method is provided for processing data in a distributed data storage system by obtaining the data and one or more corresponding checksum values from a compute node; and providing the data and the one or more corresponding checksum values to the distributed data storage system for storage. One or more checksum values corresponding to the data can be generated if the one or more checksum values are not received from a compute node. Exemplary processes are provided for copy; slice; merge: and slice and merge functions. The distributed data storage system comprises, for example, one or more Parallel Log-Structured File System (PLFS) storage elements and/or key-value storage elements storing one or more key-value pairs.