Segmented DSP Circuitry for High-Bandwidth Matrix Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Integrated circuits face challenges in efficiently performing high-bandwidth, low-latency operations for artificial intelligence, machine learning, and deep learning applications due to the significant area, power, and latency issues associated with multiplier circuits and data handling.
Innovation Solution
The design of computationally dense digital signal processing (DSP) circuitry with restructured multipliers and adders, along with cascaded DSP circuitries and tensor DSP architectures, to perform matrix multiplications and dot-product operations efficiently, reducing latency and increasing bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of multipliers is increased to accommodate growth of neural networks and machine learning applications, then computational capability is improved, but area, power, and routing resources are significantly consumed
Solution Approach 1:
The patent segments the computational workload by dividing the neural network processing into multiple stages, with each stage containing a subset of multipliers. This allows the system to achieve high computational capability through parallel processing across stages rather than requiring all multipliers to be active simultaneously, thereby reducing the area required for any single stage while maintaining overall productivity.
Solution Approach 2:
The patent introduces a temporal dimension to the computational architecture by implementing multi-stage processing where data flows through successive stages over time. This transforms the problem from a spatial arrangement (all multipliers in one large array) to a spatiotemporal arrangement (multiple smaller stages processed sequentially), reducing area consumption while maintaining computational throughput.
2Productivity
If the number of multipliers is increased to accommodate growth of neural networks and machine learning applications, then computational capability is improved, but power consumption is significantly increased
Solution Approach 1:
By segmenting the multiplier array into multiple stages that process data sequentially, the patent enables power gating of inactive stages. Only the currently active stage consumes full power, while other stages can be placed in low-power states, thereby reducing overall power consumption compared to having all multipliers continuously active.
Solution Approach 2:
The multi-stage architecture enables periodic activation of different multiplier groups. Each stage is activated in sequence for a specific time period to process its portion of the computational workload, then deactivated. This periodic action pattern reduces average power consumption while maintaining the required computational throughput.
3Productivity
If the number of multipliers is increased to accommodate growth of neural networks and machine learning applications, then computational capability is improved, but latency is introduced to computations
Solution Approach 1:
The patent segments the computational pipeline into multiple stages that can operate in a pipelined fashion. While one stage is processing data, other stages are simultaneously preparing or finalizing different portions of the computation. This segmentation enables overlapping execution of computational tasks, reducing overall latency despite the increased number of multipliers.
Solution Approach 2:
The multi-stage architecture ensures continuity of useful action by maintaining an active processing stage at all times. As one stage completes its computation and transitions to the next stage, another stage is already ready to receive and process the next data batch, eliminating idle time and maintaining continuous computational throughput, thereby reducing latency.
4Productivity
If data handling circuits are increased to handle more data for machine learning operations, then bandwidth is improved, but area and routing resources are significantly consumed
Solution Approach 1:
The data handling circuits are segmented into multiple channels corresponding to different stages of the neural network processing. Each channel handles a portion of the data flow, allowing parallel data transmission without requiring a single large-width bus. This segmented approach achieves high bandwidth through parallelism while using less area per channel compared to a monolithic data handling circuit.
Data Source
AI summary
The present disclosure relates generally to techniques for efficiently performing operations associated with artificial intelligence (AI), machine learning (ML), and/or deep learning (DL) applications, such as training and/or interference calculations, using an integrated circuit device. More specifically, the present disclosure relates to an integrated circuit design implemented to perform these operations with low latency and/or a high bandwidth of data. For example, embodiments of a computationally dense digital signal processing (DSP) circuitry, implemented to efficiently perform one or more arithmetic operations (e.g., a dot-product) on an input are disclosed. Moreover, embodiments described herein may relate to layout, design, and data scheduling of a processing element array implemented to compute matrix multiplications (e.g., systolic array multiplication).