Tensor Accelerator Normalization via Channel-Axis Convolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices face increased latency and power consumption due to memory access costs and data movement for normalization operations in neural networks, particularly when data is transferred between layers, as they rely on external processors for these operations.
Innovation Solution
Implementing convolution operations within the accelerator to perform normalization tasks, such as layer and instance normalization, without transferring data outside the accelerator, thereby reducing memory access costs and minimizing data movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is transferred between layers of neural network for normalization operations using external processors, then normalization can be performed, but latency and power consumption increase
Solution Approach 1:
The patent merges the normalization operation functionality into the accelerator itself by providing normalization operators within the accelerator. This allows normalization to be performed concurrently with convolution operations without requiring data transfer to external processors, thereby reducing latency while maintaining normalization accuracy.
Solution Approach 2:
The accelerator is designed with multi-functionality to perform both convolution operations and normalization operations (including instance normalization and layer normalization) within the same hardware unit. This eliminates the need for separate external processors and reduces data movement between layers.
2Reliability
If data is transferred between layers of neural network for normalization operations using external processors, then normalization can be performed, but power consumption increases
Solution Approach 1:
The patent merges the normalization operation functionality into the accelerator itself by providing normalization operators within the accelerator. This allows normalization to be performed concurrently with convolution operations without requiring data transfer to external processors, thereby reducing power consumption while maintaining normalization accuracy.
Solution Approach 2:
The patent extracts the normalization operation from external processors and integrates it directly into the accelerator hardware. This extraction eliminates the need for data movement to and from external processors, significantly reducing power consumption associated with memory access and data transfer.
3Reliability
If data is transferred between layers of neural network, then normalization operations can be performed by external processors, but memory access cost increases
Solution Approach 1:
The patent merges the normalization operation functionality into the accelerator itself by providing normalization operators within the accelerator. This allows normalization to be performed using on-chip memory and data already present in the accelerator, thereby reducing memory access cost while maintaining normalization accuracy.
4Productivity
If data movement is performed between accelerator and external processors for normalization, then normalization operations can be completed, but latency increases
Solution Approach 1:
The patent merges the normalization operation functionality into the accelerator itself by providing normalization operators within the accelerator. This allows normalization to be performed concurrently with convolution operations using the same data flow, eliminating the need for additional data movement and reducing latency while ensuring normalization operations are completed.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A processor-implemented tensor processing method includes: receiving a request to process a neural network including a normalization layer by an accelerator; and generating an instruction executable by the accelerator in response to the request, wherein, by executing the instruction, the accelerator is configured to determine an intermediate tensor corresponding to a result of performing a portion of operations included in the normalization layer, by performing, in a channel axis direction, a convolution based on: a target tensor on which the portion of operations is to be performed; and a kernel having a number of input channels and a number of output channels determined based on the target tensor and including elements of scaling values determined based on the target tensor.