Neural Network Inference Circuit with Multi-Core Dot Product Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional neural network computations on integrated circuits are inefficient due to the need for numerous clock cycles and resource-intensive memory operations, particularly in computing dot products, which are central to neural network operations.
Innovation Solution
The integrated circuit employs a neural network computation fabric with multiple cores and a global channel that computes dot products in parallel, aggregates partial results, and performs post-processing operations to optimize computation efficiency and reduce power consumption, utilizing ternary weight values and sparse weight sparsity to minimize circuit resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional MAC circuits are used to compute dot products sequentially, then computation accuracy is maintained, but the number of clock cycles increases and computation speed decreases
Solution Approach 1:
The patent divides the dot product computation into multiple segments, each handled by a separate processing core. Each core computes a portion of the dot product (partial sum) using segmented weight values and input values. This segmentation enables parallel computation across multiple cores, significantly reducing the total computation time while distributing the computational load to avoid excessive complexity in a single circuit.
Solution Approach 2:
The patent transitions from sequential single-core computation to multi-dimensional parallel computation by introducing multiple processing cores that operate simultaneously. The computation is distributed across spatial dimensions (multiple cores) rather than being confined to a single sequential processing path, thereby increasing computation speed without proportionally increasing overall system complexity.
2Productivity
If intermediate terms are stored in memory during dot product computation, then computation accuracy is maintained, but memory resource usage increases and computation time increases
Solution Approach 1:
The patent performs preliminary segmentation of weight values and input values before computation, organizing them into groups that can be processed in parallel. This preliminary organization allows each core to compute partial sums directly without needing to store all intermediate terms in memory, as each core only needs to retain its specific segment's data for the duration of its computation.
Solution Approach 2:
The patent creates multiple copies of the computation logic across different cores, with each core holding a subset of the weight and input values it needs. Rather than storing all intermediate computation terms in a single memory location, the data is distributed and copied to the appropriate cores that need them, reducing overall memory resource requirements while maintaining computation accuracy.
3Speed
If more processing cores are added to compute dot products in parallel, then computation speed increases, but circuit surface area increases
Solution Approach 1:
The patent segments both the computation task and the circuit architecture into multiple modular cores. Each core is designed with a standardized structure that processes a specific segment of the dot product computation. This modular segmentation allows for efficient packing of multiple cores on the chip, maximizing the use of available circuit surface area while maintaining high computation speed through parallel processing.
Solution Approach 2:
The patent designs each processing core to be universal and multi-functional, capable of handling different segments of the dot product computation. Each core can process any combination of weight and input value segments, making the cores interchangeable and highly efficient. This universality allows the same circuit pattern to be replicated multiple times, increasing computation speed while minimizing the additional surface area required compared to specialized circuits.
4Use of energy by moving object
If ternary weight values and sparse weight sparsity are used, then power consumption decreases and resource usage is minimized, but computation precision may be reduced
Solution Approach 1:
The patent changes the parameter representation of weight values from traditional binary or floating-point formats to ternary values with sparse representations. This parameter change reduces the number of bits required to represent each weight value, thereby reducing power consumption and memory resource usage. The sparsity exploitation further optimizes this by only storing and processing non-zero weight values, maintaining acceptable computation precision while significantly reducing energy consumption.
Data Source
AI summary
Some embodiments provide a method for a neural network inference circuit that executes a neural network. The method loads a first set of inputs into an input buffer and computes a first dot product between the first set of inputs and a set of weights. The method shifts the first set of inputs in the buffer while loading a second set of inputs into the buffer such that a first subset of the first set of inputs is removed from the buffer, a second subset of the first set of inputs is moved to new locations in the buffer, and a second set of inputs are loaded into locations in the buffer vacated by the shifting. The method computes a second dot product between (i) the second set of inputs and the second subset of the first set of inputs and (ii) the set of weights.