Fabric-Attached Memory Connector for HDF API Calls
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed systems face performance limitations in HDF file input/output operations due to slow access times and small bandwidths of traditional storage approaches, such as HDD-based storage and compute node-centric architectures.
Innovation Solution
The implementation of a fabric-attached memory (FAM) pool, which allows HDF files to be stored in a memory-centric architecture, enabling scalable and independent scaling of shared memory from compute nodes, and utilizing a FAM connector to handle HDF API calls through remote memory access operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional HDD-based storage or compute node-centric architectures are used, then system simplicity is maintained, but access time and bandwidth performance deteriorate
Solution Approach 1:
The system segments storage into two hierarchical levels: compute node local storage for frequent access and a centralized fabric-attached memory pool for bulk storage. This segmentation allows high-speed access for active data while maintaining overall system simplicity through clear functional separation.
Solution Approach 2:
A memory manager is introduced as an intermediary component that mediates between compute nodes and the fabric-attached memory pool. This intermediary handles the complexity of memory allocation, mapping, and coordination, enabling high-performance access without requiring compute nodes to directly manage the complex storage infrastructure.
2Productivity
If compute node-centric storage architecture is used, then ease of operation is maintained, but memory bandwidth and scalability are limited
Solution Approach 1:
The memory manager provides self-service capabilities by automatically handling memory allocation, deallocation, and mapping operations. Compute nodes can access the fabric-attached memory pool without manually managing the complex storage infrastructure, maintaining ease of operation while achieving high throughput.
Solution Approach 2:
The system transitions from a flat compute node-centric storage model to a hierarchical two-dimensional architecture with local storage and centralized memory pool. This dimensional change enables simultaneous local high-speed access and centralized resource sharing, improving throughput without proportionally increasing operational complexity.
3Adaptability or versatility
If shared memory is tightly coupled with compute nodes, then ease of operation is maintained, but adaptability and independent scaling are reduced
Solution Approach 1:
The memory architecture is segmented into compute node local memory and a separate fabric-attached memory pool. This segmentation enables independent scaling of the shared memory pool without affecting compute nodes, providing adaptability while managing complexity through clear functional separation.
Solution Approach 2:
The fabric-attached memory pool serves multiple functions: shared data storage, high-performance caching, and scalable resource allocation. This multi-functionality provides system adaptability and flexibility while the memory manager handles the architectural complexity centrally.
Data Source
AI summary
In accordance with example implementations, a process includes receiving, by a connector that is associated with a compute node and is associated with a fabric-attached memory (FAM), an application programming interface (API) called to perform an operation that is associated with a hierarchical data format (HDF) object of an HDF file. The API call includes a HDF object identifier, which corresponds to the HDF object. The process includes, responsive to the request, based on the HDF object identifier, accessing, by the connector, mapping information that is stored in the FAM; and using, by the connector, the mapping information to identify a FAM descriptor corresponding to a first data item that is stored in the FAM and corresponds to the HDF object. The process includes, responsive to the request, serving, by the connector, the API call responsive to the identification of the FAM descriptor.