Neural Network Output Bus for Layer Data Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network integrated circuits face inefficiencies in computation due to the need for numerous clock cycles and resource usage in performing dot products, particularly in convolutional neural networks, where storage and memory operations are resource-intensive and power-consuming.
Innovation Solution
The design incorporates a neural network computation fabric with multiple cores that compute partial dot products in parallel, a global channel for aggregation, and post-processing units that apply bias, scale, and activation functions efficiently, optimizing resource usage and reducing power consumption by leveraging weight sparsity and quantized outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiply-accumulate (MAC) circuits are used to compute dot products in neural networks, then computation can be performed, but numerous clock cycles are required and memory resources are heavily utilized for storing intermediate terms
Solution Approach 1:
The patent divides the dot product computation into multiple segments, each handled by a separate MAC circuit operating in parallel. Instead of computing the entire dot product sequentially in one MAC circuit, the computation is segmented into partial dot products that can be computed simultaneously, thereby reducing the total number of clock cycles required.
Solution Approach 2:
The patent introduces a spatial dimension by deploying multiple MAC circuits in parallel rather than using a single MAC circuit sequentially. This dimensional expansion from time-sequential to space-parallel computation allows multiple partial dot products to be computed simultaneously, improving computation speed without proportionally increasing the surface area of individual circuits.
2Measurement precision
If MAC circuits compute dot products term by term in separate cycles, then computation precision is maintained, but memory resources are heavily consumed for storing intermediate terms
Solution Approach 1:
The patent performs preliminary computation by calculating multiple partial dot products in parallel before final aggregation. This allows intermediate results to be computed and held in smaller, distributed buffers rather than requiring large memory resources for sequential intermediate term storage, while maintaining precision through controlled parallel computation.
Solution Approach 2:
The patent merges multiple partial dot product computations into a single aggregation step. Instead of storing each intermediate term separately in memory, the partial results from parallel MAC circuits are combined in a reduced set of buffer locations, decreasing overall memory resource consumption while preserving computational precision.
3Productivity
If parallelization is implemented to reduce clock cycles, then computation speed improves, but the surface area of the circuit must be massively expanded
Solution Approach 1:
The patent applies local quality by making each MAC circuit specialized for computing partial dot products with specific weight and input value ranges. This localization allows for optimized, compact MAC circuit designs that can be efficiently replicated in parallel without requiring massive expansion of individual circuit surface areas, as each unit is tailored to its specific computational task.
Solution Approach 2:
The patent implements dynamic resource allocation where MAC circuits and their associated buffers are activated and configured based on the specific neural network layer being processed. This dynamic approach allows the circuit to adapt its parallelization degree and resource usage to match the computational requirements of each layer, avoiding the need for massive fixed surface area expansion.
Data Source
AI summary
Some embodiments provide a neural network inference circuit (NNIC) for executing a neural network that includes multiple computation nodes at multiple layers. The NNIC includes multiple core circuits including memories for storing input values for the computation nodes. The NNIC includes a set of post-processing circuits for computing output values of the computation nodes. The output values for a first layer are for storage in the core circuits as input values for a second layer. The NNIC includes an output bus that connects the post-processing circuits to the core circuits. The output bus is for (i) receiving a set of output values from the post-processing circuits, (ii) transporting the output values of the set to the core circuits based on configuration data specifying a core circuit at which each of the output values is to be stored, and (iii) aligning the output values for storage in the core circuits.