Co-compute Unit in Lower-Level Cache for Memory-Intensive Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Memory-intensive applications, such as raytracing and machine-learning, face inefficiencies due to data exceeding the capacity of first-level caches, leading to cache thrashing and increased processing times.
Innovation Solution
Implementing co-compute units within lower-level caches, such as third-level caches, to perform operations on behalf of compute units, reducing the need for frequent data loading and eviction from first-level caches by storing necessary data in larger caches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in first-level cache for memory-intensive applications, then data access speed is improved, but cache capacity is exceeded leading to frequent data loading and eviction
Solution Approach 1:
The patent implements a nested cache hierarchy where first-level caches are embedded within compute units, second-level caches are shared across compute units, and third-level caches provide additional capacity. This nested structure allows data to be stored at multiple levels, with larger capacity caches nested outside the compute unit boundary, enabling memory-intensive applications to utilize more than just first-level cache capacity without sacrificing access speed for frequently used data.
Solution Approach 2:
The patent introduces a new dimension to the cache architecture by adding third-level caches that extend beyond the traditional two-level hierarchy. This additional cache level provides an extra dimension of storage capacity, allowing the system to accommodate larger working sets for memory-intensive applications like raytracing and machine learning without increasing first-level cache size or frequency of data loading/eviction.
2Duration of action of stationary object
If first-level cache size is increased to accommodate more data, then data retention is improved, but processing time increases due to cache thrashing
Solution Approach 1:
The patent segments the cache system into multiple independent levels: first-level caches segmented by compute unit, second-level caches segmented by groups of compute units, and third-level caches providing additional segmentation. This segmentation allows each level to retain data independently, reducing cache thrashing by preventing evictions at the first-level cache when data can be retained at lower levels in the hierarchy with longer retention durations.
Solution Approach 2:
The patent implements preliminary action by pre-loading data into second-level and third-level caches before it is needed at the first-level cache. This allows the system to prepare data in advance in larger capacity caches, reducing the frequency of data loading operations and minimizing processing time losses due to cache misses and thrashing.
3Reliability
If data is frequently loaded and evicted from first-level cache, then cache hit rate decreases, but energy consumption increases
Solution Approach 1:
The patent introduces second-level and third-level caches as intermediary storage between the first-level cache and main memory. These intermediary caches absorb the burden of frequent data loading and eviction operations, allowing first-level caches to maintain higher hit rates by keeping frequently accessed data locally. The intermediary caches handle the energy-consuming data movement operations, reducing the overall energy consumption of the system compared to frequent first-level cache thrashing.
Data Source
AI summary
A processor includes compute units each including a first-level cache and each communicatively coupled to a co-compute unit (CCU) within a lower-level cache. In response to a compute unit receiving instructions to perform operations for an application, the compute unit determines one or more parameters based on the received instructions. The compute unit then sends the parameters and instructions to perform one or more operations on behalf of the compute unit to a respective CCU. The CCU then performs the operations based on the parameters and using the lower-level cache. Once the CCU has performed the operations, the CCU then sends the results of the operations back to the compute unit.