Coarse Grain Coherency for Heterogeneous CPU GPU Memory Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graphics processing units (GPUs) face inefficiencies in handling general-purpose compute tasks due to limitations in memory coherence and programming complexity when used for heterogeneous computing environments, where both CPU and GPU need to access shared memory in a coherent manner.
Innovation Solution
Implementing a heterogeneous memory system with hardware-managed coherency between the host processor (CPU) and GPU, enabling coarse-grain coherency at a superline granularity (e.g., 1 kilobyte to 4 kilobytes) to simplify programming and reduce snoop bandwidth, using a unified virtual memory space and ownership tables to manage memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If cache-line level coherency is implemented between CPU and GPU, then memory coherence is ensured, but snoop bandwidth consumption increases and programming complexity increases
Solution Approach 1:
The patent segments the memory coherence problem by introducing multiple levels of coherence domains: fine-grained cache line level coherence for CPU-CPU interactions, and coarse-grained superline level coherence for GPU memory operations. This segmentation allows different coherence granularities to be applied to different access patterns, reducing overall system complexity while maintaining correctness.
Solution Approach 2:
The patent adds a new dimension to memory management by introducing the superline abstraction layer between cache lines and traditional memory pages. This intermediate granularity level (e.g., 4KB superlines containing multiple 64-byte cache lines) provides a middle ground that reduces snoop bandwidth while maintaining sufficient coherence for GPU operations.
2Reliability
If cache-line level coherency is implemented between CPU and GPU, then memory coherence is ensured, but snoop bandwidth consumption increases
Solution Approach 1:
The coherence system is segmented into multiple domains: CPU cache coherence operates at cache line granularity, while GPU memory coherence operates at superline granularity. This segmentation allows the system to use the larger, more efficient superline coherence domain for GPU operations, significantly reducing snoop bandwidth consumption compared to universal cache-line level coherence.
Solution Approach 2:
The patent changes the granularity parameter of memory coherence from fixed cache-line size to variable superline size (e.g., 4KB). This parameter change allows the system to optimize coherence traffic by using larger coherence units for GPU memory operations, reducing the frequency and volume of snoop requests while maintaining adequate coherence for parallel access patterns.
3Reliability
If fine-grained memory coherence is used, then data accuracy is maintained, but processing efficiency decreases
Solution Approach 1:
The patent segments the coherence enforcement scope based on access patterns: strict cache-line level coherence is maintained for CPU-CPU interactions requiring high data accuracy, while coarse-grained superline level coherence is used for GPU memory operations where approximate coherence suffices. This segmentation maintains data accuracy where critical while improving overall processing efficiency.
Solution Approach 2:
The system applies partial coherence enforcement by using superline granularity for GPU operations instead of full cache-line level coherence. This partial action approach provides sufficient coherence for most GPU workloads without the excessive overhead of fine-grained coherence, thereby improving processing efficiency while maintaining adequate data accuracy.
4Ease of operation
If hardware-managed coherency is implemented, then programming complexity is reduced, but system complexity increases
Solution Approach 1:
The patent implements hardware-managed coherency where the memory system automatically handles coherence protocols and ownership tracking without requiring software intervention. The hardware monitors memory access patterns, manages superline ownership states, and enforces coherence rules transparently, reducing programming complexity while the added hardware logic manages the increased system complexity.
Solution Approach 2:
The patent introduces hardware coherence management as an intermediary layer between CPU/GPU software and the underlying memory subsystem. This intermediary automatically handles coherence protocols, ownership tracking, and conflict resolution, shielding programmers from complexity while the hardware infrastructure manages the increased system complexity through dedicated coherence control logic.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
One embodiment provides for a general-purpose graphics processing device comprising a general-purpose graphics processing compute block to process a workload including graphics or compute operations, a first cache memory, and a coherency module enable the first cache memory to coherently cache data for the workload, the data stored in memory within a virtual address space, wherein the virtual address space shared with a separate general-purpose processor including a second cache memory that is coherent with the first cache memory.