Shuffle Manager for Distributed Memory Object Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed computing solutions fail to provide low-latency access to data objects that span memory and storage across multiple nodes, lacking support for both memory and file storage semantics, which limits performance in environments with large data sets exceeding DRAM capacity.
Innovation Solution
The MemVerge DMO system implements a distributed memory object architecture with a shuffle manager that creates sharable memory spaces using RDMA, allowing data to be stored in-memory and serialized to persistent storage when needed, enabling access as either in-memory or file-storage mode, and providing a distributed memory object sorter to manage data persistence across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in DRAM memory for fast access, then data access speed is improved, but storage capacity is limited to node memory size
Solution Approach 1:
The patent extends storage beyond the local node dimension by implementing distributed memory objects that span multiple nodes. The system creates a hierarchical storage architecture where hot data resides in local DRAM for fast access, while cold data is stored in remote nodes' memory or persistent storage, enabling capacity exceeding single-node limits while maintaining memory-speed access through RDMA.
Solution Approach 2:
The patent introduces a memory manager and shuffle manager as intermediary components that coordinate data placement and access across distributed nodes. These managers handle the complexity of tracking which data resides in which node's memory or storage, enabling transparent access to distributed data without requiring applications to manually manage storage capacity constraints.
2Reliability
If data is accessed using traditional file storage semantics, then storage durability is improved, but access latency increases
Solution Approach 1:
The patent changes the access parameter from traditional sequential file I/O to parallel memory-mapped access using RDMA. By mapping distributed memory objects into the application's address space and using direct remote memory access, the system achieves both durability (through persistence layer) and low latency (through memory-speed access), eliminating the trade-off between durability and speed.
3Quantity of substance
If distributed memory objects are implemented across multiple nodes, then storage capacity and availability are improved, but system complexity increases
Solution Approach 1:
The patent implements a universal memory manager that handles multiple functions: tracking data location across nodes, managing cache coherence, handling persistence, and coordinating access. This single multi-functional component simplifies the system by consolidating what would otherwise require multiple separate mechanisms, enabling distributed storage without proportionally increasing complexity.
4Adaptability or versatility
If nodes are dynamically added or removed from the cluster, then system adaptability is improved, but data availability and consistency become challenging
Solution Approach 1:
The patent implements a dynamic data placement system where the memory manager continuously tracks which nodes are alive and where data resides. When nodes are added or removed, the system dynamically redistributes data to maintain optimal access patterns and availability. The shuffle manager enables dynamic reshuffling of data across changing node topologies without application intervention, maintaining reliability despite cluster flexibility.
Data Source
AI summary
Disclosed herein is an apparatus and method for a shuffle manager for a distributed memory object system. In one embodiment, a method includes forming a system cluster comprising a plurality of nodes, wherein each node includes a memory, a processor and a network interface to send and receive messages and data, wherein the network interface operates on remote direct memory access; creating a plurality of sharable memory spaces having partitioned data, wherein each space is a distributed memory object having a compute node, wherein the sharable memory spaces are at least one of persistent memory or DRAM cache; and storing data in an in-memory data structure when there is available memory in a compute node; and if there is an out of memory condition, serializing at least some of the in-memory data and spilling it to a distributed memory object system to persist shuffled data outside the compute node.