Neural Network Accelerator Scatter Operation via Sparse Index Tensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network accelerators lack efficient hardware support for scatter operations, making it difficult to implement these operations efficiently, as they do not provide indexing capabilities and conventional architectures do not support scatter operations effectively.
Innovation Solution
Implementing a scatter operation in fixed-function hardware of a neural network accelerator by converting vectors of indices to sparse index tensors and generating an update tensor to update the input data tensor, using existing hardware units such as convolution engines and element-wise operations units to perform the necessary operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If scatter operations are implemented using conventional neural network accelerator architectures, then the device complexity remains low, but the productivity and processing efficiency deteriorate due to lack of hardware support
Solution Approach 1:
The scatter operation is segmented into distinct functional stages: index tensor generation from index vectors, update tensor generation by combining index tensors with update values, and application of updates to the input tensor. Each stage can be implemented using separate hardware units already present in the neural network accelerator, avoiding the need for a monolithic scatter operation unit.
Solution Approach 2:
Existing hardware units in the neural network accelerator (convolution engines, element-wise operation units, matrix multiplication units) are made multi-functional by configuring them to perform scatter operation tasks. For example, convolution engines can generate index tensors from index vectors, and element-wise operation units can combine index tensors with update values to produce update tensors.
2Productivity
If external hardware units like GPUs or CPUs are used to perform scatter operations, then the processing efficiency improves, but the device complexity and dependency on external resources increases
Solution Approach 1:
The scatter operation functionality is merged with the existing neural network accelerator hardware. Instead of having separate external units (GPU/CPU) handle scatter operations, the same hardware units that perform neural network computations are configured to also perform scatter operations, consolidating functionality within a single device.
Solution Approach 2:
The neural network accelerator serves itself by implementing scatter operations using its own internal hardware units. The accelerator does not need to offload scatter operations to external resources; instead, it uses its existing computational units (convolution engines, matrix multiplication units) to generate index tensors, combine them with update values, and apply updates to input tensors.
Data Source
AI summary
A method of implementing a scatter operation in fixed-function hardware of a neural network accelerator involves converting two or more vectors of indices to sparse index tensors in a one-hot sparse format. An update tensor is generated, by applying the update values to one of the sparse index tensors (or a tensor derived from it). In some examples, an input data tensor is updated based on the update tensor. In other examples, the update tensor itself is output.