Burst Buffer Appliance Small File Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In high-performance computing environments, the concurrent generation of numerous small data files by multiple compute nodes leads to excessive file system access, undermining IO throughput performance.
Innovation Solution
A burst buffer appliance is introduced between the computer system and the file system, utilizing high-speed memory and an analytics engine to aggregate small data files into containers with metadata, reducing the need for direct file system access and implementing a virtual layer of a parallel log-structured file system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple compute nodes generate numerous small data files concurrently, then data processing capability is improved, but file system access overhead increases significantly
Solution Approach 1:
The patent merges multiple small data files from different compute nodes into a single container object. This consolidation reduces the number of individual file system access operations, as the container is treated as a single unit rather than multiple separate files, thereby reducing file system access overhead while maintaining the ability to process multiple data files concurrently.
Solution Approach 2:
The patent introduces a container as an intermediary layer between compute nodes and the file system. This container acts as a mediator that absorbs and manages multiple small file operations, preventing them from directly overwhelming the file system. The container structure allows the system to handle numerous small files through a simplified interface.
2Reliability
If small data files are stored directly in the file system, then data persistence is achieved, but IO throughput performance deteriorates
Solution Approach 1:
The patent combines multiple small data files into a single container object that is then stored in the file system. This merging approach ensures data persistence while improving IO throughput, as the file system handles fewer, larger container objects rather than numerous small files. The container structure preserves all original data files while presenting a simplified storage interface.
3Adaptability or versatility
If numerous small data files are accessed, then data availability is improved, but system performance is undermined
Solution Approach 1:
The container serves as an intermediary structure that maintains data availability for multiple files while improving system performance. The container object provides a unified access point that preserves access to all contained data files, yet reduces the performance penalty associated with accessing numerous individual files through the file system interface.
Data Source
AI summary
A burst buffer appliance is adapted for coupling between a computer system and a file system. The burst buffer appliance comprises a flash memory or other high-speed memory having a substantially lower access time than the file system, and an analytics engine, and is configured to aggregate multiple data files having a size below a designated threshold into a container having at least one associated metadata file for processing by the analytics engine. The burst buffer appliance directs storage of the container in the file system. The analytics engine is configured to generate at least a content-based portion of the one or more metadata files of the container based at least in part on content of one or more of the multiple data files so as to permit one or more subsequent processing operations to be performed using the metadata file(s) rather than the corresponding multiple data files.