Distributed Memory Object Snapshot via Generation Tags
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for distributed computing environments fail to provide low-latency access to multi-node distributed data objects with both memory and file storage semantics, lacking cross-node replication and availability.
Innovation Solution
The MemVerge DMO system creates a distributed memory object architecture that enables low-latency memory spaces accessible across a cluster of nodes, exceeding memory capacity, and spanning memory and storage, using persistent memory and DRAM cache with a generation tag and doubly linked list for snapshot creation and replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If persistent memory is used for direct access to achieve memory-speed data access, then access latency is reduced, but cross-node replication and availability are not provided
Solution Approach 1:
The system segments data into key-value pairs stored in persistent memory on individual nodes, while maintaining separate metadata structures that track replication status across nodes. This allows each node to provide fast local access to its persistent memory while the distributed metadata layer ensures cross-node availability through replication.
Solution Approach 2:
The patent introduces an intermediary metadata structure that mediates between the persistent memory storage layer and the access layer. This metadata layer tracks which nodes hold copies of which key-value pairs, enabling fast local access to persistent memory while coordinating replication across nodes to ensure availability.
2Quantity of substance
If distributed memory objects are created across multiple nodes to exceed single-node memory capacity, then storage capacity is increased, but access latency increases due to network communication
Solution Approach 1:
The system implements local quality by storing frequently accessed key-value pairs in persistent memory on the local node, while less frequently accessed data is distributed to other nodes. This creates a quality gradient where locally available data has higher access priority and faster access speed, while distributed data provides additional capacity.
Solution Approach 2:
The patent adds a hierarchical dimension to data storage by organizing data across multiple levels: local persistent memory for fast access, node-level caching structures, and distributed storage across nodes. This dimensional organization allows the system to provide both high-speed local access and extended capacity through hierarchical data distribution.
3Reliability
If snapshot replication is implemented to provide availability, then data reliability is improved, but system complexity increases due to snapshot management and index regeneration
Solution Approach 1:
The system performs preliminary actions by creating snapshots of key-value store states at specific points in time before failures or data loss events. These snapshots are stored and managed in advance, allowing for quick restoration and replication without complex real-time coordination, thus reducing operational complexity while improving reliability.
Solution Approach 2:
The patent uses copying by creating replicated copies of key-value pairs across multiple nodes and maintaining snapshot copies of the entire key-value store state. These copies are managed through simplified replication protocols that duplicate data without requiring complex synchronization mechanisms, reducing system complexity while enhancing availability.
4Stability of the object's composition
If data is stored in persistent memory with generation tags and doubly linked lists for snapshots, then consistency is improved, but memory usage and data structure complexity increase
Solution Approach 1:
The system implements periodic action by using generation tags that increment periodically to mark snapshot boundaries. Instead of continuous complex tracking, the system uses discrete generation markers that periodically capture consistent states, simplifying the consistency mechanism while maintaining stability through these periodic snapshots.
Solution Approach 2:
The patent applies parameter changes by using generation tags as a simple integer parameter to track consistency states, rather than complex data structures. The doubly linked list uses straightforward next/previous pointers that link snapshots together, changing the state representation from complex to simple parameter-based tracking while maintaining data consistency.
Data Source
AI summary
Disclosed herein is an apparatus and method for a key value store snapshot for a distributed memory object system. In one embodiment, a method includes forming a system cluster comprising a plurality of nodes, wherein each node includes a memory, a processor and a network interface to send and receive messages and data; creating a plurality of sharable memory spaces having partitioned data, wherein each space is a distributed memory object having a compute node, wherein the sharable memory spaces are at least one of persistent memory or DRAM cache; storing data in persistent memory, the data having a generation tag created from a generation counter and a doubly linked list having a current view and a snapshot view, the data further being stored in either a root or a persisted row; creating a snapshot comprising a consistent point-in-time view of key value contents within a node and incrementing the generation counter; copying the snapshot to a second node; regenerating an index for the key value contents within the node; and logging updates since the snap was applied to update copied data in the second node.