Neural Network Inference Circuit Dot Product Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network computations on integrated circuits are inefficient due to the need for numerous clock cycles and resource-intensive memory operations in performing dot products, particularly in convolutional neural networks, which require significant circuit resources and power.
Innovation Solution
An integrated circuit design that includes a neural network computation fabric with multiple cores and a global channel for parallel dot product computations, post-processing operations, and efficient memory management, utilizing ternary weight values and sparse data representation to reduce resource usage and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiply-accumulate (MAC) circuits are used to compute dot products, then computation can be performed, but numerous clock cycles are required and memory resources are consumed for storing intermediate terms
Solution Approach 1:
The patent segments the dot product computation into multiple partial dot products, where each core computes a separate partial dot product using a subset of weight values. These partial results are then aggregated to form the complete dot product. This segmentation enables parallel computation across multiple cores, significantly reducing the total computation time and clock cycles required compared to sequential MAC operations.
2Reliability
If numerous memory operations are performed for dot product computation, then intermediate terms can be stored, but resource usage and power consumption increase
Solution Approach 1:
The patent extracts and utilizes the sparsity property of neural network weight matrices by identifying and computing only the non-zero weight values. By taking out only the relevant non-zero elements for computation and ignoring zero values, the system reduces memory read operations, computation operations, and associated power consumption while maintaining computation accuracy.
3Productivity
If parallelization is implemented to speed up computation, then processing throughput increases, but circuit surface area expands
Solution Approach 1:
The patent implements dynamic configuration of the computation fabric, where the system can adaptively allocate and reconfigure computational resources based on the specific computation requirements. The computation fabric can be dynamically configured to perform different operations (dot products, element-wise operations, reductions) and can adjust the number of active cores and their organization, enabling efficient parallelization without permanently expanding circuit surface area for all possible parallel operations.
4Adaptability or versatility
If standard dot product computation is used in convolutional neural networks, then neural network operations can be performed, but significant circuit resources are consumed
Solution Approach 1:
The patent creates a universal computation fabric that can perform multiple types of neural network operations including dot products, element-wise additions, element-wise multiplications, and reduction operations. By designing a multi-functional computation unit that can execute various operations, the system reduces overall circuit complexity compared to having separate dedicated circuits for each operation type, while maintaining full adaptability for different neural network layer requirements.
Data Source
AI summary
Some embodiments provide a neural network inference circuit for executing a neural network that includes multiple layers of computation nodes. At least a subset of the layers include non-convolutional layers. The neural network inference circuit includes multiple cores with memories that store input values for the layers. The cores are grouped into multiple clusters. For each cluster, the neural network inference circuit includes a set of processing circuits for receiving input values from the cores of the cluster and executing the computation nodes of the non-convolutional layers.