High Bandwidth Memory Crossbar Switch for Neural Network Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware platforms face challenges in achieving efficient memory access and performance for neural network processing due to limitations in memory access and layout schemes, which are critical for handling computationally and data-intensive artificial intelligence tasks.
Innovation Solution
A high-bandwidth memory system is designed with multiple memory units surrounding a processing component, each connected via multiple network connections, allowing parallel processing and dynamic distribution schemes to optimize memory access. This system includes request processing units that decompose memory requests into partial requests and route them through a crossbar switch to appropriate memory banks, enabling efficient data transfer and utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a traditional memory access scheme is used, then the memory system is simple to implement, but the memory access performance and bandwidth are insufficient for neural network processing
Solution Approach 1:
The memory system is segmented into multiple independent memory units (first memory unit, second memory unit, etc.), each capable of handling requests independently. This segmentation allows parallel processing of memory requests, significantly improving memory access performance and bandwidth while maintaining manageable complexity through modular design
Solution Approach 2:
The patent introduces a crossbar switch that creates a two-dimensional connectivity structure between processing elements and memory units. This dimensional change enables multiple simultaneous access paths, transforming the traditional single-path memory access into a multi-path parallel access system, thereby improving performance without linearly increasing complexity
2Adaptability or versatility
If fixed memory distribution is used, then the memory layout is simple, but the system cannot adapt to different neural network processing requirements
Solution Approach 1:
The memory distribution scheme is made dynamic and programmable, allowing the system to adapt to different neural network processing requirements. The crossbar switch can be reconfigured to implement different distribution patterns (e.g., row-major, column-major, or custom patterns), providing versatility while keeping the underlying hardware structure relatively simple
Solution Approach 2:
The system changes the distribution parameters programmatically to adapt to different workloads. By modifying distribution parameters (such as stride, block size, or mapping patterns) rather than changing the physical memory architecture, the system achieves high adaptability with minimal increase in hardware complexity
3Productivity
If memory requests are processed sequentially, then the control logic is simple, but the memory bandwidth is limited
Solution Approach 1:
The request processing function is segmented and distributed across multiple memory units, each with its own control logic. This allows simultaneous processing of multiple requests in parallel, dramatically increasing memory bandwidth. The segmentation of control logic into distributed units manages complexity by localizing processing decisions at each memory unit
Solution Approach 2:
Multiple memory requests are merged into a single broadcast operation that reaches all memory units simultaneously. The crossbar switch combines multiple request streams and routes them to appropriate memory units in parallel, achieving high bandwidth while keeping the control logic at each memory unit relatively simple through request consolidation
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system comprises a processor coupled to a plurality of memory units. Each of the plurality of memory units includes a request processing unit and a plurality of memory banks. Each request processing unit includes a plurality of decomposition units and a crossbar switch, the crossbar switch communicatively connecting each of the plurality of decomposition units to each of the plurality of memory banks. The processor includes a plurality of processing elements and a communication network communicatively connecting the plurality of processing elements to the plurality of memory units. At least a first processing element of the plurality of processing elements includes a control logic unit and a matrix compute engine. The control logic unit is configured to access the plurality of memory units using a dynamically programmable distribution scheme.