Distributed Shared Memory via Metadata Service Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Sharing computer memory resources across multiple systems in a network is challenging due to hardware limitations and requires significant architectural changes and specialized software, making it difficult to efficiently combine and manage memory capacity.
Innovation Solution
A distributed shared memory system is implemented, where a clustered memory cache is aggregated from physically distinct computing systems, managed by local memory managers, and accessed via a network, using metadata services and policy managers to coordinate access, insertion, and eviction of data items, with support for Remote Direct Memory Access (RDMA) and various policies for cache management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If discrete memory chips are combined using specialized bus circuits and tight coupling, then memory capacity can be increased, but hardware requirements become complex and architectural changes are required
Solution Approach 1:
The memory system is segmented into distributed memory regions across multiple independent nodes, each with its own memory controller. This allows memory capacity to be scaled by adding nodes without requiring complex centralized bus circuits, as each node operates independently with standard network communication.
Solution Approach 2:
A metadata service acts as an intermediary between clients and distributed memory regions. The metadata service manages the mapping and coordination of memory access across nodes, eliminating the need for specialized bus circuits while maintaining efficient memory access through standard network protocols.
2Adaptability or versatility
If specialized bus circuits are used to combine memory chips, then memory sharing is enabled, but significant architectural changes and specialized software are required
Solution Approach 1:
The system uses standard network infrastructure and protocols that can serve multiple purposes beyond memory sharing, including data transfer, communication, and coordination. This universal approach enables memory sharing without requiring specialized architectures, as the same network infrastructure handles both memory access and general-purpose data communication.
Solution Approach 2:
Physical bus circuits are replaced with network-based communication protocols. Instead of dedicated electrical connections between memory chips, the system uses standard network packets and protocols to transmit memory data, eliminating the need for specialized hardware architectures while maintaining memory sharing functionality.
3Quantity of substance
If memory is distributed across multiple nodes with local memory controllers, then memory capacity and scalability are increased, but coordination and management complexity increases
Solution Approach 1:
The metadata service implements feedback mechanisms to track memory usage, access patterns, and node status across the distributed system. This feedback enables automatic load balancing, memory allocation, and coordination without manual intervention, managing the complexity of distributed memory while maintaining scalability and efficient resource utilization.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
A computer network with distributed shared memory, including a clustered memory cache aggregated from and comprised of physical memory locations on a plurality of physically distinct computing systems. The network also includes a plurality of local cache managers, each of which are associated with a different portion of the clustered memory cache, and a metadata service operatively coupled with the local cache managers. Also, a plurality of clients are operatively coupled with the metadata service and the local cache managers. In response to a request issuing from any of the clients for a data item present in the clustered memory cache, the metadata service is configured to respond with identification of the local cache manager associated with the portion of the clustered memory cache containing such data item.