MESL Memory Coherence Protocol for CPU-GPU Traffic Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing data volume due to AI and ML applications exceeds the compute power capabilities of microprocessors, leading to high memory coherence traffic between CPUs and GPUs, necessitating a solution that maintains memory coherence without relying on remote procedure calls (RPCs).
Innovation Solution
A memory coherence protocol, referred to as the MESL protocol, is implemented to track the state and epoch number of memory pages across nodes, allowing for efficient ownership management and access control, thereby reducing the need for RPCs and minimizing traffic between nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RPCs are used to maintain tight memory coherence between CPUs and GPUs, then memory coherence is maintained, but network traffic between nodes increases significantly
Solution Approach 1:
The patent extracts the memory coherence management functionality from the RPC mechanism and implements it as a separate, optimized protocol. By taking out the coherence management tasks from the general-purpose RPC system, the invention creates a specialized path for coherence operations that generates less network traffic while maintaining the required reliability of memory coherence between CPU and GPU nodes.
Solution Approach 2:
The patent introduces an intermediary coherence management system that mediates between the CPU and GPU nodes. This intermediary layer handles memory coherence requests through optimized protocols, reducing the need for high-volume RPC traffic while ensuring memory coherence is maintained. The intermediary acts as a specialized controller that manages coherence state transitions without requiring full RPC overhead.
2Power
If microprocessors are used for compute tasks, then system complexity is low, but compute power is insufficient for AI and ML workloads
Solution Approach 1:
The patent merges CPU and GPU nodes into a unified distributed system with shared memory coherence protocols. By combining the processing capabilities of CPUs with the accelerated compute power of GPUs in a coordinated multi-node architecture, the system achieves high compute power for AI and ML workloads while managing complexity through standardized coherence protocols that abstract the underlying heterogeneity.
Solution Approach 2:
The patent creates a universal memory coherence protocol that works across different node types (CPU and GPU). This multi-functional protocol handles coherence requirements for various workload types and hardware configurations, enabling the system to scale compute power by adding different node types without proportionally increasing system complexity. The universal protocol acts as a common interface that manages diversity.
Data Source
AI summary
A coherence protocol applied to memory pages maintains coherence between memory spaces on a plurality of nodes so that the threads of the runtime are operable on any of the nodes. The nodes operating according to the coherence protocol track a state and an epoch number for each memory page residing therein. The states include a modified state in which only one particular node has an up-to-date copy of the memory page, an exclusive state in which only one particular node owns the memory page, a shared state in which all nodes that have the memory page in the shared state have the same copy, and a lost state in which the memory page cannot be either read or written. The epoch number is a number that is incremented each time the page enters the modified state and is used to determine whether the page contains data that is stale.