Erasure Metadata Precomputation on Compute Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large scale compute architectures face bottlenecks due to compute nodes waiting for storage systems to complete input/output operations, particularly during computationally intensive tasks like creating parity metadata, leading to idle computational capacity.
Innovation Solution
Computing parity metadata, such as erasure codes, using the computational capacities of compute nodes, and precomputing data layout to match storage system performance characteristics, allowing direct writing to storage using RDMA operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the storage system computes parity metadata and performs data layout organization, then the storage system can maintain data integrity and optimize storage performance, but the compute nodes experience increased waiting time and lose valuable compute time during I/O operations
Solution Approach 1:
The patent extracts the parity metadata computation function from the storage system and relocates it to the compute nodes. Compute nodes independently calculate erasure code parity metadata for their local data buffers, eliminating the need to wait for storage system processing and removing the bottleneck that caused compute time loss while maintaining data integrity through distributed parity calculation
Solution Approach 2:
The patent implements preliminary computation of parity metadata by compute nodes on their local data buffers before data is transferred to the storage system. This advance computation ensures parity information is ready when needed, preventing compute nodes from idling during storage operations while ensuring data can be properly reconstructed if needed
2Productivity
If compute nodes perform computationally intensive metadata processing tasks, then storage system throughput is improved, but compute nodes have reduced computational capacity for primary compute functions
Solution Approach 1:
The patent segments the metadata processing workload by having each compute node independently handle only its own data buffer's parity computation. This distributes the computational load across multiple nodes rather than concentrating it, allowing parallel processing that improves overall storage throughput while each node maintains capacity for its primary compute functions
Solution Approach 2:
Compute nodes serve themselves by independently computing their own parity metadata without requiring storage system resources. This self-service approach eliminates the bottleneck where storage system processing limited throughput, while compute nodes use their idle computational capacity during I/O wait periods rather than reducing capacity for primary functions
3Productivity
If the storage system reorganizes application write data to match storage performance characteristics, then storage performance is optimized, but this requires additional processing time and computational resources at the storage system
Solution Approach 1:
The patent implements preliminary data layout computation where the system determines the optimal storage layout before data is written. By querying the storage system in advance to determine the anticipated layout and pre-organizing data accordingly, the system optimizes storage performance without requiring time-consuming reorganization operations during the actual write process
Data Source
AI summary
Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.