Flexible Compute Array Utilization in Tensor Processors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning accelerators face performance bottlenecks in edge devices with strict power consumption constraints, particularly in applications like AR/VR, due to inefficient utilization of compute arrays during convolutional neural network operations, especially when dealing with input tensors of varying sizes and shapes.
Innovation Solution
A method for flexible compute array utilization in tensor processors, where instructions are generated to optimize data loading and processing based on the dimensions of input tensors, allowing for contiguous storage and retrieval of activation elements and weights, and dynamic configuration of subarrays to match the processing capacity, enabling efficient parallelization across multiple dimensions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If existing ML accelerators use high compute parallelism to speed up convolutional layers, then processing speed is improved, but power consumption increases beyond constraints of edge devices
Solution Approach 1:
The compute array is segmented into multiple subarrays, each capable of independent operation. This allows the system to activate only the necessary number of subarrays based on the input tensor size, reducing power consumption while maintaining processing throughput for smaller tensors commonly used in edge device inference
Solution Approach 2:
The system dynamically configures the number and arrangement of active subarrays based on the dimensions of the input tensor. This dynamic adaptation allows the accelerator to optimize the balance between processing speed and power consumption for different workloads, preventing full-array activation for small tensors
2Productivity
If existing ML accelerators use fixed compute array configuration, then hardware design is simplified, but utilization efficiency decreases for varying tensor sizes
Solution Approach 1:
The compute array configuration is made dynamic through programmable control of subarray activation and data flow routing. The system can reconfigure which subarrays are active and how data is distributed to them based on input tensor dimensions, achieving high utilization efficiency without requiring completely different hardware for each tensor size
Solution Approach 2:
The same compute array infrastructure serves multiple functions by accommodating different tensor sizes and shapes through dynamic subarray configuration. A single hardware design handles various convolution scenarios by selectively activating appropriate subarrays, eliminating the need for multiple specialized configurations
3Productivity
If existing ML accelerators process small input tensors with full compute array, then processing capacity is maximized, but energy efficiency decreases for edge device inference
Solution Approach 1:
By segmenting the compute array into smaller subarrays, the system can activate only the necessary portion for small tensor processing. This prevents energy waste from activating the full array when only a fraction of compute units are needed, directly improving energy efficiency for edge device inference workloads
Solution Approach 2:
The system changes the operational parameters of the compute array by adjusting the number of active subarrays based on input tensor size. For small tensors, fewer subarrays are activated, changing the effective processing capacity to match the workload and improve energy efficiency
Data Source
AI summary
In one embodiment, a method for machine learning acceleration includes receiving instructions to perform convolution on an input tensor using a filter tensor, determining that the size of a first dimension of the input tensor is less than a processing capacity of each of multiple subarrays of computation units in a tensor processor, selecting a second dimension of the input tensor along which to perform the convolution, selecting, based on the second dimension, one or more dimensions of the filter tensor, generating (1) first instructions for reading, using vector read operations, activation elements in the input tensor organized such that elements with different values in the second dimension are stored contiguously in memory, and (2) second instructions for reading weights of the filter tensor along the selected one or more dimensions, and using the first and second instructions to provide the activation elements and the weights to the subarrays.