Hardware Cache Logical Ports for Off-Chip Memory Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hardware accelerators face performance bottlenecks due to high latency when accessing data stored in off-chip memories like DDR and HBM, as traditional caching mechanisms fail to adequately address these latency issues.
Innovation Solution
A scalable hardware cache with a fully-associative design and true LRU replacement policy, featuring configurable logical ports and thread management, including a request interface, thread processing scheduler, and LRU eviction manager, to optimize memory access and minimize off-chip memory accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional caching mechanisms are used, then the system structure remains simple, but the latency associated with accessing off-chip memory data remains high
Solution Approach 1:
The caching system is segmented into multiple independent cache banks (first cache bank, second cache bank, third cache bank) that can operate in parallel. Each cache bank handles specific data types or access patterns, allowing simultaneous processing of multiple memory access requests and reducing overall latency without requiring a single complex cache controller.
Solution Approach 2:
The patent introduces a new dimension to the caching architecture by adding temporal and spatial multiplexing through multiple cache banks and support for both cached and non-cached access modes. This dimensional expansion allows the system to handle diverse memory access patterns concurrently, reducing latency without proportionally increasing controller complexity.
2Speed
If multiple cache banks are introduced to reduce latency, then memory access speed improves, but the device complexity increases
Solution Approach 1:
Each cache bank is designed with multi-functionality to handle different types of memory access requests (cached and non-cached) and support various data types. The unified interface and common controller architecture allow the same hardware structure to serve multiple purposes, increasing access speed without linearly increasing overall system complexity.
Solution Approach 2:
The patent merges multiple cache banks under a unified controller architecture with shared interfaces and common control logic. This consolidation allows parallel operation of multiple banks while avoiding the complexity of completely independent cache systems, achieving high speed access with managed complexity through resource sharing.
3Loss of time
If cached access mode is used for all requests, then cache hit rate may increase, but latency for non-cached data access worsens
Solution Approach 1:
The system dynamically selects between cached and non-cached access modes based on the specific data access request and cache state. The controller can switch access modes on-the-fly without fixed configuration, optimizing for cache hits when applicable while maintaining efficient pathways for non-cached access, thus balancing latency reduction with overall throughput preservation.
Solution Approach 2:
The patent changes the access mode parameter dynamically based on request characteristics and cache status. By adjusting whether a request uses cached or non-cached mode according to real-time conditions, the system achieves low latency for cacheable data while maintaining productivity for non-cached data through optimized access pathways.
Data Source
AI summary
Systems and methods for a scalable hardware cache with configurable logical ports and related thread management are described. A scalable hardware cache includes a request interface having a first logical port and a second logical port associated with a fully-associative cache memory. The first logical port is configured to receive a first set of read requests with an expected cache hit and the second logical port is configured to receive a second set of read requests with an expected cache miss. The scalable hardware cache further includes thread processing circuitry to manage a first maximum number of a first set of threads for processing the first set of read requests and a second maximum number of a second set of threads for processing the second set of read requests that can be active at a given time based on a performance metric associated with the scalable hardware cache.