Virtual Burst Buffer Filesystem for HPC Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current parallel filesystems and burst buffer technologies face limitations such as the need for costly specialized hardware, static configurations, and inability to reconfigure without downtime, leading to inefficiencies and resource wastage in high-performance computing environments.
Innovation Solution
A system and method for creating an on-demand virtual intermediary filesystem with virtual burst buffers that can be dynamically created and managed within a compute node cluster, utilizing existing storage devices without the need for expensive SSDs or NVRAMs, allowing for real-time allocation and deallocation of resources based on application needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If costly specialized HPS devices (SSDs or NVRAMs) are used to build burst buffers, then data transfer performance is improved, but system cost increases significantly
Solution Approach 1:
The patent creates a virtual copy of the burst buffer functionality using software-based virtualization. Instead of physical HPS devices, it uses virtual burst buffers that map to existing storage devices, replicating the high-performance interface through virtual file systems and namespace mapping while avoiding the cost of specialized hardware
Solution Approach 2:
The patent replaces the mechanical/physical burst buffer hardware system with a software-based virtual file system. The virtual burst buffer uses virtualization techniques to substitute physical HPS devices with software layers that manage data transfer between compute nodes and storage systems, eliminating the need for specialized hardware components
2Stability of the object's composition
If parallel filesystems are created statically in advance, then system stability is improved, but adaptability to changing application needs deteriorates
Solution Approach 1:
The patent implements a dynamic parallel file system that can be created, modified, and destroyed on-demand. The virtual burst buffer configuration changes dynamically based on application requirements, allowing the system to adapt filesystem parameters such as size, location, and performance characteristics in real-time without requiring static pre-configuration
Solution Approach 2:
The virtual file system layer provides universal access to burst buffer functionality across multiple compute nodes and storage devices. It creates a unified namespace that can map to various underlying storage configurations, making the system adaptable to different application needs while maintaining a consistent interface
3Adaptability or versatility
If burst buffer architecture is reconfigured to meet new requirements, then adaptability is improved, but system downtime increases
Solution Approach 1:
The patent pre-allocates storage space on compute nodes for potential virtual burst buffer usage before applications need it. The virtual file system infrastructure is prepared in advance, allowing rapid creation and configuration of virtual burst buffers when applications require them, eliminating reconfiguration downtime
4Speed
If dedicated high-performance storage devices are allocated for burst buffers, then data transfer speed is improved, but storage device utilization deteriorates due to unused capacity
Solution Approach 1:
The patent merges the burst buffer storage capacity with application working storage on compute nodes. The same storage devices serve dual purposes: as virtual burst buffers for high-speed data transfer and as working storage for applications, eliminating unused capacity while maintaining high transfer rates through virtual file system management
Solution Approach 2:
The virtual burst buffer system automatically manages storage allocation and deallocation based on application needs. It dynamically identifies and utilizes available storage capacity on compute nodes, allowing the system to self-optimize storage utilization without dedicated unused hardware, while maintaining high-performance data transfer capabilities
Data Source
AI summary
A method for creating a virtual intermediary filesystem on the fly and in real-time on a compute node cluster having a plurality of compute nodes that are running or scheduled to run an application job is provided The method includes initializing a data structure, allocating a plurality of compute nodes, reserving the compute node cluster, distributing the application job, determining memory utilization, determining potential free memory space, marking the potential free memory space, and mapping the marked potential free memory space to create the virtual intermediary filesystem on-demand and in real-time, wherein the virtual intermediary filesystem is arranged to store data in parallel and concurrently from a plurality of computing devices on the compute node cluster and output the stored data to a storage system, wherein the computing devices output the data at a rate that is greater than a maximum write or read data rate of the storage system.