Parallel Layer-Norm Circuit for DNN Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for layer normalization in deep neural networks are energy-inefficient and suffer from high latency due to their reliance on microprocessors and memory utilization, which constrains throughput and delays processing in subsequent layers.
Innovation Solution
An integrated circuit and system architecture that utilize parallel processing to efficiently compute mean and standard deviation across large vectors, employing a digital circuit and circuit blocks to perform layer normalization in analog memory-based artificial neural networks, reducing latency and improving throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If conventional methods using microprocessors and memory are used for layer normalization, then general-purpose computing is achieved, but energy efficiency deteriorates and latency increases
Solution Approach 1:
The layer normalization operation is divided into multiple parallel stages: (1) computing partial sums and partial sums of squares in parallel across multiple circuit blocks, (2) aggregating these partial results to compute global mean and variance, and (3) computing the final normalized output. This segmentation enables concurrent processing that reduces latency while maintaining energy efficiency through specialized hardware for each stage.
Solution Approach 2:
The patent replaces conventional microprocessor-based computation with specialized analog memory-based circuitry. The crossbar array and circuit blocks perform mathematical operations (multiplication, addition, squaring) directly in hardware, eliminating the need for data movement between memory and processor, thereby reducing energy consumption and latency.
2Productivity
If conventional memory utilization methods are used, then data storage is achieved, but throughput is constrained due to data movement requirements
Solution Approach 1:
The patent merges storage and computation functions by implementing the layer normalization operation directly within the memory crossbar array. The input data remains in the crossbar array while computation operations are performed in-place, eliminating data movement between separate storage and processing units. This merging enables high throughput by allowing multiple operations to proceed concurrently without memory access bottlenecks.
Solution Approach 2:
The memory crossbar array performs computation operations autonomously on the data stored within it. The circuit blocks and crossbar array work together to compute sums, sums of squares, means, and variances directly on the resident data without requiring external processor intervention, thereby eliminating processing delays associated with data transfer.
3Loss of time
If parallel processing is implemented using specialized hardware, then latency is reduced and throughput is improved, but device complexity increases
Solution Approach 1:
The crossbar array and circuit blocks are designed to perform multiple functions: they can store input data, compute partial sums, compute partial sums of squares, aggregate results, and generate normalized output. This multi-functionality reduces the need for separate dedicated hardware for each operation, thereby managing device complexity while maintaining parallel processing benefits for reducing latency.
Data Source
AI summary
Systems and methods for performing layer normalization are described. A circuit can receive a sequence of input data across a plurality of clock cycles, where the sequence of input data represents a portion of an input vector. The circuit can determine a plurality of sums and a plurality of sums of squares corresponding to the sequence of input data. The circuit can determine, based on the plurality of sums of squares, a first scalar representing an inverse square-root of a variance of vector elements in the input vector. The circuit can determine a second scalar representing a negation of a product of the first scalar and a mean of the vector elements in the input vector. The circuit can determine, based on the first scalar, the second scalar and the received sequence of input data, an output vector that is a normalization of the input vector.