Unified Memory Cluster via Software-Defined Storage Controller
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems face limitations in data access performance and scalability, particularly in information processing systems with multiple compute nodes, where conventional solutions like SAP HANA and Tachyon struggle with node capacity and latency.
Innovation Solution
Implementing an in-memory database using software-defined storage functionality, which creates a unified memory cluster across compute nodes, allowing for dynamic expansion and improved data access through a software-defined storage controller like ScaleIO, and integrating it as an additional tier in a multi-tier storage system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional in-memory database implementations (SAP HANA, Tachyon) are used, then data access performance is maintained, but the number of supported nodes is limited and scalability is poor
Solution Approach 1:
The system divides the in-memory database into distributed segments across multiple compute nodes, with each node managing a portion of the database. The storage system is segmented into multiple storage tiers (fast tier, capacity tier) that can be independently managed and scaled. This segmentation allows the system to support a larger number of nodes while maintaining manageable complexity at each node.
Solution Approach 2:
A software-defined storage controller is introduced as an intermediary layer between compute nodes and physical storage devices. This controller abstracts the complexity of storage management, providing unified access to the distributed in-memory database across multiple nodes while handling data distribution, replication, and tiering operations centrally.
2Speed
If data is stored in conventional storage tiers, then storage capacity is adequate, but data access latency increases for frequently accessed data
Solution Approach 1:
Frequently accessed data is pre-loaded into the in-memory database on fast storage tiers before actual access occurs. The system proactively manages data movement between tiers based on access patterns, ensuring that hot data is already in high-speed memory when needed, thereby eliminating access latency.
Solution Approach 2:
The system implements different storage qualities in different locations: hot data resides in high-speed in-memory storage on the fast tier with low latency characteristics, while cold data is stored on the capacity tier with higher capacity but slower access. This local quality differentiation optimizes both speed and latency for their respective use cases.
3Productivity
If memory resources are allocated to each compute node independently, then node autonomy is maintained, but overall data access performance and clustering efficiency deteriorate
Solution Approach 1:
The system merges the memory resources of multiple compute nodes into a unified in-memory database cluster. Individual node memories are combined to form a distributed memory pool that provides shared access to all nodes, improving data access performance through localized memory access while maintaining the benefits of clustering.
Solution Approach 2:
The software-defined storage controller provides universal management functionality for the distributed memory cluster, handling data distribution, replication, tiering, and access coordination across all compute nodes. This universal controller enables each node to participate in the clustered memory system without requiring node-specific configuration or management complexity.
4Adaptability or versatility
If storage system capacity is increased to support more nodes, then node scalability improves, but data access latency increases
Solution Approach 1:
The system adds a temporal dimension to storage capacity management by implementing multi-tier storage with different access speeds. As the system scales to support more nodes and larger data volumes, it automatically manages data placement across time-sensitive (fast tier) and capacity-oriented (capacity tier) storage, allowing capacity to scale without proportionally increasing latency for active data.
Data Source
AI summary
An apparatus in one embodiment comprises a plurality of compute nodes each having a processor coupled to a memory. The apparatus further comprises a storage system configured for communication with the compute nodes over at least one network, with the storage system comprising a software-defined storage controller configured to implement an in-memory database accessible as a unified memory cluster to respective processors of the compute nodes. The in-memory database comprises portions of respective memories of the compute nodes. The in-memory database may be expanded, for example, using an additional portion of the memory of one of the compute nodes, responsive to an overflow event associated with the memory of another one of the compute nodes. The compute nodes may illustratively comprise respective analysis nodes that run respective applications configured to utilize analytics functionality of the in-memory database. Other illustrative embodiments include systems, methods and processor-readable storage media.