Active NVRAM Sub-Processor for HPC Data Post-Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High volume data processing in High Performance Computing (HPC) leads to bottlenecks in the data staging process and I/O traffic between processors and storage systems, necessitating efficient post-processing solutions.
Innovation Solution
The implementation of active Non-Volatile Random Access Memory (NVRAM) components with a sub-processing unit and queue management system to perform post-processing of data chunks, relieving main processing cores and enabling efficient data handling and checkpoint mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data post-processing is performed using main processing cores in HPC clusters, then data can be reformatted, sorted, or compressed before storage, but bottlenecks occur in the data staging process and I/O traffic between processors and storage systems
Solution Approach 1:
The patent divides the post-processing function into separate processing units that operate independently from the main HPC processing cores. Each processing unit is dedicated to specific post-processing tasks (reformatting, sorting, compression) and can handle data chunks in parallel, thereby segmenting the overall data flow to eliminate bottlenecks in the staging process.
Solution Approach 2:
The patent introduces intermediate processing units that act as mediators between the main HPC processing cores and the storage systems. These intermediary units buffer and pre-process data before it reaches the storage system, reducing I/O traffic delays and allowing the main cores to continue processing without waiting for storage operations.
2Quantity of substance
If data is stored across multiple nodes in persistent storage systems, then data capacity and redundancy are improved, but I/O traffic between processors and storage systems increases causing bottlenecks
Solution Approach 1:
The patent performs preliminary post-processing actions (compression, formatting, sorting) on data chunks before they are distributed to persistent storage nodes. By preparing data in advance at the processing stage rather than during storage operations, the system reduces the I/O burden when writing to multiple storage nodes, thereby maintaining high productivity despite increased storage capacity requirements.
3Reliability
If main processing cores perform post-processing of data chunks, then data can be prepared for storage, but the processing burden on main cores increases reducing overall system efficiency
Solution Approach 1:
The patent segments the processing system into distinct functional units: main HPC processing cores for computational tasks, and separate dedicated processing units for post-processing operations. This segmentation ensures that data processing completeness is maintained while distributing the processing burden appropriately, preventing overload of the main cores and reducing overall system complexity through clear functional separation.
Data Source
AI summary
A computing node includes an active Non-Volatile Random Access Memory (NVRAM) component which includes memory and a sub-processor component. The memory is to store data chunks received from a processor core, the data chunks comprising metadata indicating a type of post-processing to be performed on data within the data chunks. The sub-processor component is to perform post-processing of said data chunks based on said metadata.