Neural Network Inference Circuit Dynamic Memory Sleep
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional neural network integrated circuits face inefficiencies due to the need for numerous clock cycles and resource utilization in computing dot products, particularly in storing intermediate terms, which slows down computations and increases power consumption.
Innovation Solution
The integration of a neural network computation fabric with cores that compute dot products and a global channel for aggregation, along with dynamic memory sleep capabilities and optimized circuit design, such as using ternary weight values and discrete quantization, to enhance efficiency and reduce power usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If MAC circuits are used to compute dot products sequentially, then computation accuracy is maintained, but computation speed decreases and power consumption increases
Solution Approach 1:
The patent divides the neural network computation into multiple segments or layers, with each layer processed by dedicated hardware circuits. This segmentation enables parallel processing of different layers simultaneously, dramatically improving computation speed while reducing per-cycle power consumption compared to sequential MAC operations
Solution Approach 2:
The patent replaces traditional sequential MAC circuit operations with a hardware-based parallel computation architecture. Instead of using software-controlled sequential multiplication and accumulation, the invention uses dedicated hardware circuits that perform dot product computations in parallel, achieving both speed improvement and energy efficiency
2Productivity
If intermediate terms are stored in memory during computation, then computation accuracy is maintained, but memory resource usage increases and computation time increases
Solution Approach 1:
The patent loads and stores neural network parameters (weights, biases, activation functions) into hardware circuits before computation begins. This preliminary action eliminates the need for continuous memory access during computation, as all necessary data is pre-loaded into the computation fabric, thereby improving productivity while reducing actual memory resource usage during operation
Solution Approach 2:
The patent introduces a computation fabric as an intermediary between traditional memory and processing units. This fabric includes specialized hardware circuits that hold and process neural network parameters locally, acting as a buffer that reduces frequent memory access requirements while maintaining computation accuracy
3Measurement precision
If full precision floating-point operations are used, then computation accuracy is maintained, but power consumption increases and computation speed decreases
Solution Approach 1:
The patent changes the numerical precision parameters from full floating-point to lower-precision formats (such as integer or fixed-point representations). This parameter change is sufficient for neural network computations where extreme precision is not required, thereby improving computation speed while maintaining adequate accuracy for the application
Data Source
AI summary
For a neural network inference circuit that executes a neural network including multiple computation nodes at multiple layers for which data is stored in a plurality of memory banks, some embodiments provide a method for dynamically putting memory banks into a sleep mode of operation to conserve power. The method tracks the accesses to individual memory banks and, if a certain number of clock cycles elapse with no access to a particular memory bank, sends a signal to the memory bank indicating that it should operate in a sleep mode. Circuit components involved in dynamic memory sleep, in some embodiments, include a core RAM pipeline, a core RAM sleep controller, a set of core RAM bank select decoders, and a set of core RAM memory bank wrappers.