Tensor Controller Architecture for Edge ML Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning accelerators face performance bottlenecks in convolutional neural networks due to high compute requirements and data movement, particularly in edge devices with strict power consumption constraints, such as AR/VR headsets and mobile devices, where they struggle to efficiently process convolutional layers in real-time.
Innovation Solution
A multi-level control architecture for machine learning accelerators, featuring tensor processor clusters with shared controllers and address generators, that optimize data parallelism and memory address generation by using counter stacks and dependency controllers to manage traversal sizes and token dependencies, enabling efficient memory address generation and computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing ML accelerators use high compute parallelism and optimized data orchestration, then processing speed of convolutional layers is improved, but power consumption increases making them unsuitable for edge devices
Solution Approach 1:
The system segments the feature map traversal into multiple independent counter stacks, each responsible for a specific dimension. This segmentation allows parallel processing while reducing the power consumption of individual counter units, as each counter operates independently with smaller scope rather than one large parallel processing unit consuming high power.
Solution Approach 2:
The counter stacks dynamically increment based on traversal sizes and are controlled by dependency tokens that signal when data is ready. This dynamic control allows the system to activate only the necessary computing resources for current workloads, avoiding the continuous high power consumption of static parallel processing architectures.
2Productivity
If ML accelerators are designed for high performance computation, then convolutional layer processing capability is improved, but adaptability to real-time inference in resource-constrained environments deteriorates
Solution Approach 1:
The system uses dynamic counter incrementation controlled by dependency tokens, allowing the accelerator to adapt its computation pace to real-time data availability and workload requirements. This dynamic behavior enables the system to function effectively in resource-constrained edge environments while maintaining high computation capability when resources are available.
Solution Approach 2:
Dependency tokens provide feedback mechanisms that signal when input data, weights, or output buffers are ready. This feedback control allows the system to synchronize computation with data availability, improving real-time adaptability without sacrificing computational throughput by ensuring operations only proceed when necessary data is prepared.
3Speed
If counter stacks increment without dependency control, then memory address generation speed is improved, but data access correctness deteriorates due to premature access
Solution Approach 1:
Dependency tokens provide feedback signals that control when counters should increment. The first token triggers activation counter increments, the second token triggers weight counter increments, and the third token triggers output counter increments. This feedback mechanism ensures counters only advance when corresponding data is ready, maintaining address generation speed while ensuring data access correctness through conditional control.
Solution Approach 2:
The system prepares dependency tokens in advance to signal readiness of data, weights, and output buffers before counter incrementation occurs. This preliminary action ensures that by the time counters generate memory addresses, the required data is already prepared and available, preventing premature access while maintaining efficient address generation flow.
Data Source
AI summary
A machine-learning accelerator system, comprising: a plurality of controllers each configured to traverse a feature map with n-dimensions according to instructions that specify, for each of the n-dimensions, a respective traversal size, wherein each controller comprises: a counter stack comprising counters each associated with a respective dimension of the n-dimensions of the feature map, wherein each counter is configured to increment a respective count from a respective initial value to the respective traversal size associated with the respective dimension associated with that counter; a plurality of address generators each configured to use the respective counts of the counters to generate at least one memory address at which a portion of the feature map is stored; and a dependency controller computing module configured to (1) track conditional statuses for incrementing the counters and (2) allow or disallow each of the counters to increment based on the conditional statuses.