Unified Memory Cluster for Multi-Tier Storage Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems face challenges in data access performance and scalability, particularly in information processing systems with large numbers of compute nodes, where conventional storage tier implementations struggle to provide optimal performance and support a large number of nodes.
Innovation Solution
Implementing a multi-tier storage system with a front-end storage tier as a unified memory cluster using software-defined storage functionality, where the unified memory cluster is accessible to processors of multiple compute nodes and can be formed from portions of their memories or associated IO nodes, allowing for distributed software-defined storage modules to manage data access efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional storage tier implementations are used, then system structure is simple, but data access performance and scalability are limited
Solution Approach 1:
The storage system is segmented into distinct tiers: front-end storage tier using unified memory cluster for high-performance data access, back-end storage tier for capacity, and software-defined storage controller for management. This segmentation allows each tier to be optimized independently, improving overall data access performance while maintaining manageable complexity through clear functional separation.
Solution Approach 2:
A software-defined storage controller acts as an intermediary between compute nodes and storage resources, managing the unified memory cluster and coordinating data access operations. This intermediary layer abstracts the complexity of multi-tier storage management, enabling high performance without requiring compute nodes to directly manage complex storage operations.
2Adaptability or versatility
If conventional storage tier implementations are used, then device complexity is low, but scalability to large numbers of nodes is limited
Solution Approach 1:
The unified memory cluster serves multiple functions: it acts as both the front-end storage tier for high-performance access and as shared memory resources across multiple compute nodes. The software-defined storage controller provides universal management capabilities that can handle any compute node joining the system, enabling scalability without proportionally increasing complexity.
Solution Approach 2:
The system transitions from traditional hierarchical storage access to a unified memory space dimension, where compute nodes access storage through a shared memory cluster rather than through multiple hierarchical layers. This dimensional change enables linear scalability as nodes can be added to the unified memory space without fundamentally changing the access architecture.
3Loss of time
If unified memory cluster is implemented across multiple nodes, then data access latency is reduced, but system complexity increases
Solution Approach 1:
Memory resources from multiple compute nodes are merged into a unified memory cluster that functions as a single shared storage tier. This merging provides low-latency access comparable to local memory while distributing capacity across nodes. The software-defined storage controller manages the merged resources, hiding the complexity of coordination and data placement from individual nodes.
4Adaptability or versatility
If software-defined storage controller is used, then storage capacity can be dynamically modified, but control complexity increases
Solution Approach 1:
The software-defined storage controller implements dynamic storage capacity management by allowing memory resources to be allocated and de-allocated from compute nodes based on demand. Storage capacity can be modified without hardware changes - the controller dynamically reconfigures the unified memory cluster by adding or removing nodes and adjusting memory allocations, providing adaptability while the software layer manages the complexity of these dynamic changes.
Data Source
AI summary
An apparatus in one embodiment comprises a multi-tier storage system having at least a front-end storage tier and a back-end storage tier. The storage system comprises a software-defined storage controller configured to implement the front-end storage tier as a unified memory cluster accessible to respective processors of a plurality of compute nodes. The software-defined storage controller is implemented using at least one processing device comprising a processor coupled to a memory. The unified memory cluster in some embodiments comprises portions of respective memories of at least a subset of the compute nodes. The portions of respective memories of at least a subset of the compute nodes may comprise, for example, respective flash memories of those compute nodes, or respective dynamic random access memories of those compute nodes. Other illustrative embodiments include systems, methods and processor-readable storage media.