HBM Bridge Controllers for Direct Memory Access to Processing Elements
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High bandwidth memory (HBM) chips are not utilized to their full capacity due to limitations in data retrieval to centralized caches, leading to increased processing times and reduced bandwidth in accelerator architectures.
Innovation Solution
A bridge controller coordinates memory access requests and distributes data values directly to processing elements, optimizing data retrieval by connecting each channel of the HBM to a designated set of processing elements, using direct memory access (DMA) descriptors in alternating sets to enhance data retrieval efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is retrieved through a centralized cache, then data distribution is simplified, but bandwidth utilization is reduced and processing time increases
Solution Approach 1:
The patent segments the centralized cache architecture into distributed cache memory across multiple processing elements. Each processing element has its own cache memory, allowing parallel data access and eliminating the single bottleneck of a centralized cache. This segmentation enables full bandwidth utilization of the HBM by allowing multiple processing elements to access data simultaneously without contending for a single cache resource.
Solution Approach 2:
The patent introduces bridge circuits as intermediary components that directly connect HBM channels to processing elements. These bridge circuits eliminate the need for data to traverse through a centralized cache, providing a direct data path that reduces processing time and maximizes bandwidth utilization. The bridge acts as a mediator that enables efficient data transfer between memory and processing elements without the overhead of centralized cache management.
2Speed
If a centralized cache is used for data retrieval, then data management is simplified, but data transmission paths become longer
Solution Approach 1:
The patent segments the data retrieval path by distributing cache memory to individual processing elements. This eliminates the long transmission path from HBM to centralized cache and then to processing elements, replacing it with shorter direct paths from HBM through bridge circuits to individual processing elements. Each processing element accesses data from its local cache or directly from HBM, significantly reducing transmission distance and increasing speed.
Solution Approach 2:
The patent changes the data access architecture from a hierarchical dimension (HBM → centralized cache → processing elements) to a parallel dimension where multiple processing elements can access HBM simultaneously through dedicated bridge circuits. This dimensional change from sequential hierarchical access to parallel direct access reduces both transmission path length and retrieval time.
3Productivity
If HBM is connected to centralized cache, then system architecture is simplified, but full bandwidth capability of HBM cannot be utilized
Solution Approach 1:
The patent segments the HBM connection architecture by providing dedicated bridge circuits for each HBM channel that connect directly to multiple processing elements. This segmentation allows each HBM channel to be fully utilized by multiple processing elements simultaneously, achieving full bandwidth capability. The increased architectural complexity of multiple bridges is offset by the significant gain in bandwidth utilization and processing throughput.
Data Source
AI summary
An example device includes a bank of processing elements; a high bandwidth memory module in communication with the bank of processing elements and including a plurality of channels of memory; a plurality of bridges corresponding to the plurality of channels of memory, each bridge configured to connect a designated channel of the channels of memory to a designated vector of processing elements in the bank and including a bridge controller configured to: in response to a request for a memory access for a processing operation, perform the memory access to retrieve a data value from the designated channel according to the request; and provide the data value to a processing element in the designated vector to process according to the processing operation.