Early Termination Mechanism for Artificial Neural Network Layer Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks (ANNs) face challenges in reducing power consumption and latency due to fixed processing durations, as they typically execute until predefined completion times, leading to unnecessary operations and potential accuracy issues from excessive processing.
Innovation Solution
An early termination mechanism is implemented in the NN processor, which sorts weights by output and input functions, detects saturation in MAC outputs, and triggers early termination when calculations become redundant, saving power and improving latency by terminating layers before completion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If fixed processing duration is used in ANN, then processing completeness is ensured, but power consumption increases and latency increases
Solution Approach 1:
The patent applies dynamics by transitioning from fixed processing duration to dynamic early termination. The system monitors saturation indicators during processing and adapts the processing duration based on whether saturation is detected, allowing the system to terminate processing early when saturation occurs while ensuring completeness when saturation does not occur.
Solution Approach 2:
The patent implements feedback by continuously monitoring saturation indicators during neural network processing. The saturation state is detected and fed back to the processing control mechanism, which then decides whether to terminate processing early or continue, creating a closed-loop control system that optimizes power consumption based on actual processing needs.
2Reliability
If fixed processing duration is used in ANN, then processing completeness is ensured, but latency increases
Solution Approach 1:
The system dynamically adjusts processing duration based on saturation detection. When saturation is detected, processing is terminated early, reducing latency. When saturation is not detected, processing continues to ensure completeness, maintaining reliability. This dynamic approach resolves the contradiction between fixed duration and variable needs.
Solution Approach 2:
The patent applies partial action by performing only the necessary processing steps required to achieve saturation or determine its absence. Instead of always completing full processing duration, the system performs partial processing when saturation occurs early, reducing latency while maintaining processing effectiveness.
3Measurement precision
If processing continues until predefined completion, then accuracy is maintained, but unnecessary operations increase power consumption
Solution Approach 1:
The system uses feedback from saturation indicators to determine when processing can be stopped. When saturation is detected, it provides feedback that accuracy has been achieved, allowing early termination. This prevents energy waste from unnecessary operations while maintaining accuracy through condition-based termination.
Solution Approach 2:
The neural network processing system serves itself by autonomously detecting saturation and self-terminating when accuracy is achieved. The system monitors its own state and makes termination decisions without external intervention, eliminating energy waste from predefined completion requirements.
4Reliability
If excessive processing is performed, then completeness is ensured, but statistical confidence decreases due to over-amplified indicators
Solution Approach 1:
The patent applies partial action by performing processing only until saturation is detected, avoiding excessive processing. This prevents over-amplification of indicators that would occur with excessive processing, maintaining statistical confidence while ensuring processing completeness through saturation-based termination criteria.
Data Source
AI summary
A novel and useful system and method of early termination for use in an artificial neural network (ANN). An NN processor incorporates the early termination mechanism that provides the capability of terminating a compute graph in a data flow architecture, e.g., an ANN, earlier than its predefined planned execution. This serves to improve both power consumption and sometimes latency considering the additional operations that are not performed when the network is terminated early. The early termination mechanism is implemented partly in the SDK/compiler offline at compile time and partly at runtime in the NN processor. During compile time, the weights of the neural network are sorted first by output function and then by input function. In operation, the LCU receives feedback from the MAC units in the processing elements (PEs) and if saturation in the MAC outputs is detected and crosses a threshold, it means the calculations performed until that point are sufficient and that additional calculations are not likely to change the results significantly. Thus, early termination for that layer can be triggered thereby saving power and improving latency.