Accelerator Tensor Processing for Neural Network Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices face inefficiencies in performing normalization operations within neural networks, as they often require data movement between accelerators and host processors, increasing latency and power consumption.
Innovation Solution
Implementing a method where the accelerator performs convolution operations to achieve normalization layer results without external data transmission, by generating kernels based on input tensors and scaling values to compute intermediate tensors equivalent to normalization outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If normalization operations are performed using external host processors, then computational flexibility is maintained, but data movement between accelerator and host processor increases latency and power consumption
Solution Approach 1:
The patent merges the normalization operation functionality into the accelerator by implementing convolution operations that can perform both filtering and normalization tasks. The accelerator integrates the ability to compute mean and variance directly within its convolution engine, eliminating the need for separate normalization processing on the host processor. This combining of functions reduces data movement while maintaining computational capability through unified processing pipelines.
2Speed
If data is transmitted between accelerator and host processor for normalization operations, then operational simplicity is maintained, but processing speed decreases due to increased latency
Solution Approach 1:
The accelerator performs self-service by implementing its own normalization capabilities through convolution operations. The device computes mean and variance directly within the accelerator using its existing computational resources, without requiring external host processor intervention. The convolution engine serves dual purposes: traditional filtering and normalization computation, allowing the system to handle normalization operations autonomously and efficiently.
3Productivity
If convolution operations are used to perform normalization, then data movement is reduced, but the complexity of kernel generation and operation increases
Solution Approach 1:
The system performs preliminary action by pre-generating kernels with appropriate scaling values before the actual normalization operation. The kernel generation process is prepared in advance, where scaling factors are computed and embedded into the kernel structure beforehand. This preliminary preparation simplifies the runtime operation, as the accelerator only needs to execute the pre-configured convolution with the pre-computed kernel, rather than performing complex calculations during inference.
Data Source
AI summary
A processor-implemented tensor processing method includes: receiving a request to process a neural network including a normalization layer by an accelerator; and generating an instruction executable by the accelerator in response to the request, wherein, by executing the instruction, the accelerator is configured to determine an intermediate tensor corresponding to a result of a portion of operations of the normalization layer, by performing, in a channel axis direction, a convolution based on an input tensor and a kernel, wherein the input tensor is of the normalization layer and includes a plurality of channels, a number of input channels of the kernel is determined based on the input tensor, and scaling values of elements of the kernel are determined based on the number of input channels.